Control transfer of a vehicle

ABSTRACT

A method for detecting obstacles, the method may include receiving, from a plurality of vehicles, and by an I/O module of a computerized system, visual information acquired during executions of vehicle maneuvers that are suspected as being obstacle avoidance maneuvers; determining, based at least on the visual information, at least one visual obstacle identifier for visually identifying at least one obstacle; and transmitting to one or more of the plurality of vehicles, the at least one visual obstacle identifier.

CROSS REFERENCE

The application is a continuation of PCT patent applicationPCT/IB2019058208 filing date Sep. 27, 2019 which claims priority fromU.S. provisional Ser. No. 62/747,147 filing date Oct. 18, 2018. Thisapplication claims priority from U.S. provisional patent Ser. No.62/827,112 filing date Mar. 31, 2019 both incorporated herein byreference.

TECHNICAL FIELD

The present disclosure generally relates to detecting and avoidingobstacles in an autonomous driving environment.

BACKGROUND

Assisted and autonomous driving systems are known in the art. In suchsystems, computer implemented systems control (at least to some extent)some, or all, of a vehicle's driving functions, e.g., speed, telemetry,braking, etc. The vehicle is typically equipped with one or moresensors, e.g., a camera, to provide the system with current informationregarding the driving environment. The current information for thedriving environment is typically used by the driving system to determinehow to drive on roadways according to road maps stored on the vehicle.

SUMMARY

A method for detecting obstacles, the method may include receiving, froma plurality of vehicles, and by an I/O module of a computerized system,visual information acquired during executions of vehicle maneuvers thatare suspected as being obstacle avoidance maneuvers; determining, basedat least on the visual information, at least one visual obstacleidentifier for visually identifying at least one obstacle; andtransmitting to one or more of the plurality of vehicles, the at leastone visual obstacle identifier.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the disclosure will be understood and appreciatedmore fully from the following detailed description, taken in conjunctionwith the drawings in which:

FIG. 1A illustrates an example of a method;

FIG. 1B illustrates an example of a signature;

FIG. 1C illustrates an example of a dimension expansion process;

FIG. 1D illustrates an example of a merge operation;

FIG. 1E illustrates an example of hybrid process;

FIG. 1F illustrates an example of a first iteration of the dimensionexpansion process;

FIG. 1G illustrates an example of a method;

FIG. 1H illustrates an example of a method;

FIG. 1I illustrates an example of a method;

FIG. 1J illustrates an example of a method;

FIG. 1K illustrates an example of a method;

FIG. 1L illustrates an example of a method;

FIG. 1M illustrates an example of a method;

FIG. 1N illustrates an example of a matching process and a generation ofa higher accuracy shape information;

FIG. 1O illustrates an example of an image and image identifiers;

FIG. 1P illustrates an example of an image, approximated regions ofinterest, compressed shape information and image identifiers;

FIG. 1Q illustrates an example of an image, approximated regions ofinterest, compressed shape information and image identifiers;

FIG. 1R illustrates an example of an image;

FIG. 1S illustrates an example of a method;

FIG. 2A illustrates an example of images of different scales;

FIG. 2B illustrates an example of images of different scales;

FIG. 2C illustrates an example of a method;

FIG. 2D illustrates an example of a method;

FIG. 2E illustrates an example of a method;

FIG. 2F illustrates an example of a method;

FIG. 2G illustrates an example of different images;

FIG. 2H illustrates an example of a method;

FIG. 2I illustrates an example of a method;

FIG. 2J illustrates an example of a method;

FIG. 2K illustrates an example of different images acquisition angles;

FIG. 2L illustrates an example of a method;

FIG. 2M illustrates an example of a method;

FIG. 2N illustrates an example of a system;

FIG. 3A is a partly-pictorial, partly-block diagram illustration of anexemplary obstacle detection and mapping system, constructed andoperative in accordance with embodiments described herein;

FIG. 3B is a block diagram of an exemplary autonomous driving system tobe integrated in the vehicle of FIG. 3A;

FIG. 3C is a flowchart of an exemplary process to be performed by theautonomous driving system of FIG. 3B;

FIG. 4 is a block diagram of an exemplary obstacle avoidance server ofFIG. 3A;

FIG. 5 is a flowchart of an exemplary process to be performed by theobstacle avoidance server of FIG. 4;

FIG. 6 is an example of a method;

FIG. 7 is an example of a method;

FIG. 8 is an example of a driving scenario;

FIG. 9 is an example of a driving scenario;

FIG. 10 is an example of a driving scenario;

FIG. 11 is an example of a method;

FIG. 12 is an example of a method;

FIG. 13 is an example of a method;

FIG. 14 is an example of a driving scenario;

FIG. 15 is an example of a driving scenario;

FIG. 16 is an example of a scene;

FIG. 17 is an example of a scene;

FIG. 18 is an example of a driving scenario;

FIG. 19 is an example of a method;

FIG. 20 is an example of a method;

FIG. 21 is an example of a method;

FIG. 22 is an example of a driving scenario;

FIG. 23 is an example of a driving scenario;

FIG. 24 is an example of a driving scenario;

FIG. 25 is an example of a driving scenario;

FIG. 26 is an example of a driving scenario;

FIG. 27 is an example of a driving scenario;

FIG. 28 is an example of a method;

FIG. 29 is an example of entity movement functions;

FIG. 30 is an example of a method;

FIG. 31 is an example of a method;

FIG. 32 is an example of a method;

FIG. 33 is an example of a method;

FIG. 34 is an example of a method;

FIG. 35 is an example of a method;

FIG. 36 is an example of a driving scenario;

FIG. 37 is an example of a method;

FIG. 38 is an example of a method;

FIGS. 39-44 illustrate various data structures including a concept, testimages and matching results, as well as various processes related to thedata structures;

FIG. 45 illustrates an example of a system;

FIG. 46 is an example of a method; and

FIG. 47 is an example of a method.

DESCRIPTION OF EXAMPLE EMBODIMENTS

The specification and/or drawings may refer to an image. An image is anexample of a media unit. Any reference to an image may be appliedmutatis mutandis to a media unit. A media unit may be an example ofsensed information. Any reference to a media unit may be applied mutatismutandis to a natural signal such as but not limited to signal generatedby nature, signal representing human behavior, signal representingoperations related to the stock market, a medical signal, and the like.Any reference to a media unit may be applied mutatis mutandis to sensedinformation. The sensed information may be sensed by any type ofsensors—such as a visual light camera, or a sensor that may senseinfrared, radar imagery, ultrasound, electro-optics, radiography, LIDAR(light detection and ranging), etc.

The specification and/or drawings may refer to a processor. Theprocessor may be a processing circuitry. The processing circuitry may beimplemented as a central processing unit (CPU), and/or one or more otherintegrated circuits such as application-specific integrated circuits(ASICs), field programmable gate arrays (FPGAs), full-custom integratedcircuits, etc., or a combination of such integrated circuits.

Any combination of any steps of any method illustrated in thespecification and/or drawings may be provided.

Any combination of any subject matter of any of claims may be provided.

Any combinations of systems, units, components, processors, sensors,illustrated in the specification and/or drawings may be provided.

Low Power Generation of Signatures

The analysis of content of a media unit may be executed by generating asignature of the media unit and by comparing the signature to referencesignatures. The reference signatures may be arranged in one or moreconcept structures or may be arranged in any other manner. Thesignatures may be used for object detection or for any other use.

The signature may be generated by creating a multidimensionalrepresentation of the media unit. The multidimensional representation ofthe media unit may have a very large number of dimensions. The highnumber of dimensions may guarantee that the multidimensionalrepresentation of different media units that include different objectsis sparse—and that object identifiers of different objects are distantfrom each other—thus improving the robustness of the signatures.

The generation of the signature is executed in an iterative manner thatincludes multiple iterations, each iteration may include an expansionoperations that is followed by a merge operation. The expansionoperation of an iteration is performed by spanning elements of thatiteration. By determining, per iteration, which spanning elements (ofthat iteration) are relevant—and reducing the power consumption ofirrelevant spanning elements—a significant amount of power may be saved.

In many cases, most of the spanning elements of an iteration areirrelevant—thus after determining (by the spanning elements) theirrelevancy—the spanning elements that are deemed to be irrelevant may beshut down a/or enter an idle mode.

FIG. 1A illustrates a method 5000 for generating a signature of a mediaunit.

Method 5000 may start by step 5010 of receiving or generating sensedinformation.

The sensed information may be a media unit of multiple objects.

Step 5010 may be followed by processing the media unit by performingmultiple iterations, wherein at least some of the multiple iterationscomprises applying, by spanning elements of the iteration, dimensionexpansion process that are followed by a merge operation.

The processing may include:

Step 5020 of performing a k'th iteration expansion process (k may be avariable that is used to track the number of iterations).

Step 5030 of performing a k'th iteration merge process.

Step 5040 of changing the value of k.

Step 5050 of checking if all required iterations were done—if soproceeding to step 5060 of completing the generation of the signature.Else—jumping to step 5020.

The output of step 5020 is a k'th iteration expansion results 5120.

The output of step 5030 is a k'th iteration merge results 5130.

For each iteration (except the first iteration)—the merge result of theprevious iteration is an input to the current iteration expansionprocess.

At least some of the K iterations involve selectively reducing the powerconsumption of some spanning elements (during step 5020) that are deemedto be irrelevant.

FIG. 1B is an example of an image signature 6027 of a media unit that isan image 6000 and of an outcome 6013 of the last (K'th) iteration.

The image 6001 is virtually segments to segments 6000(i,k). The segmentsmay be of the same shape and size but this is not necessarily so.

Outcome 6013 may be a tensor that includes a vector of values per eachsegment of the media unit. One or more objects may appear in a certainsegment. For each object—an object identifier (of the signature) pointsto locations of significant values, within a certain vector associatedwith the certain segment.

For example—a top left segment (6001(1,1)) of the image may berepresented in the outcome 6013 by a vector V(1,1) 6017(1,1) that hasmultiple values. The number of values per vector may exceed 100, 200,500, 1000, and the like.

The significant values (for example—more than 10, 20, 30, 40 values,and/or more than 0.1%, 0.2%. 0.5%, 1%, 5% of all values of the vectorand the like) may be selected. The significant values may have thevalues—but may be selected in any other manner.

FIG. 1B illustrates a set of significant responses 6015(1,1) of vectorV(1,1) 6017(1,1). The set includes five significant values (such asfirst significant value SV1(1,1) 6013(1,1,1), second significant valueSV2(1,1), third significant value SV3(1,1), fourth significant valueSV4(1,1), and fifth significant value SV5(1,1) 6013(1,1,5).

The image signature 6027 includes five indexes for the retrieval of thefive significant values—first till fifth identifiers ID1-ID5 are indexesfor retrieving the first till fifth significant values.

FIG. 1C illustrates a k'th iteration expansion process.

The k'th iteration expansion process start by receiving the mergeresults 5060′ of a previous iteration.

The merge results of a previous iteration may include values areindicative of previous expansion processes—for example—may includevalues that are indicative of relevant spanning elements from a previousexpansion operation, values indicative of relevant regions of interestin a multidimensional representation of the merge results of a previousiteration.

The merge results (of the previous iteration) are fed to spanningelements such as spanning elements 5061(1)-5061(J).

Each spanning element is associated with a unique set of values. The setmay include one or more values. The spanning elements apply differentfunctions that may be orthogonal to each other. Using non-orthogonalfunctions may increase the number of spanning elements—but thisincrement may be tolerable.

The spanning elements may apply functions that are decorrelated to eachother—even if not orthogonal to each other.

The spanning elements may be associated with different combinations ofobject identifiers that may “cover” multiple possible media units.Candidates for combinations of object identifiers may be selected invarious manners—for example based on their occurrence in various images(such as test images) randomly, pseudo randomly, according to some rulesand the like. Out of these candidates the combinations may be selectedto be decorrelated, to cover said multiple possible media units and/orin a manner that certain objects are mapped to the same spanningelements.

Each spanning element compares the values of the merge results to theunique set (associated with the spanning element) and if there is amatch—then the spanning element is deemed to be relevant. If so—thespanning element completes the expansion operation.

If there is no match—the spanning element is deemed to be irrelevant andenters a low power mode. The low power mode may also be referred to asan idle mode, a standby mode, and the like. The low power mode is termedlow power because the power consumption of an irrelevant spanningelement is lower than the power consumption of a relevant spanningelement.

In FIG. 1C various spanning elements are relevant (5061(1)-5061(3)) andone spanning element is irrelevant (5061(J)).

Each relevant spanning element may perform a spanning operation thatincludes assigning an output value that is indicative of an identity ofthe relevant spanning elements of the iteration. The output value mayalso be indicative of identities of previous relevant spanning elements(from previous iterations).

For example—assuming that spanning element number fifty is relevant andis associated with a unique set of values of eight and four—then theoutput value may reflect the numbers fifty, four and eight—for exampleone thousand multiplied by (fifty+forty) plus forty. Any other mappingfunction may be applied.

FIG. 1C also illustrates the steps executed by each spanning element:

Checking if the merge results are relevant to the spanning element (step5091).

If-so—completing the spanning operation (step 5093).

If not—entering an idle state (step 5092).

FIG. 1D is an example of various merge operations.

A merge operation may include finding regions of interest. The regionsof interest are regions within a multidimensional representation of thesensed information. A region of interest may exhibit a more significantresponse (for example a stronger, higher intensity response).

The merge operation (executed during a k'th iteration merge operation)may include at least one of the following:

Step 5031 of searching for overlaps between regions of interest (of thek'th iteration expansion operation results) and define regions ofinterest that are related to the overlaps.

Step 5032 of determining to drop one or more region of interest, anddropping according to the determination.

Step 5033 of searching for relationships between regions of interest (ofthe k'th iteration expansion operation results) and define regions ofinterest that are related to the relationship.

Step 5034 of searching for proximate regions of interest (of the k'thiteration expansion operation results) and define regions of interestthat are related to the proximity. Proximate may be a distance that is acertain fraction (for example less than 1%) of the multi-dimensionalspace, may be a certain fraction of at least one of the regions ofinterest that are tested for proximity.

Step 5035 of searching for relationships between regions of interest (ofthe k'th iteration expansion operation results) and define regions ofinterest that are related to the relationship.

Step 5036 of merging and/or dropping k'th iteration regions of interestbased on shape information related to shape of the k'th iterationregions of interest.

The same merge operations may applied in different iterations.

Alternatively, different merge operations may be executed duringdifferent iterations.

FIG. 1E illustrates an example of a hybrid process and an input image6001.

The hybrid process is hybrid in the sense that some expansion and mergeoperations are executed by a convolutional neural network (CNN) and someexpansion and merge operations (denoted additional iterations ofexpansion and merge) are not executed by the CNN—but rather by a processthat may include determining a relevancy of spanning elements andentering irrelevant spanning elements to a low power mode.

In FIG. 1E one or more initial iterations are executed by first andsecond CNN layers 6010(1) and 6010(2) that apply first and secondfunctions 6015(1) and 6015(2).

The output of these layers provided information about image properties.The image properties may not amount to object detection. Imageproperties may include location of edges, properties of curves, and thelike.

The CNN may include additional layers (for example third till N'th layer6010(N)) that may provide a CNN output 6018 that may include objectdetection information. It should be noted that the additional layers maynot be included.

It should be noted that executing the entire signature generationprocess by a hardware CNN of fixed connectivity may have a higher powerconsumption—as the CNN will not be able to reduce the power consumptionof irrelevant nodes.

FIG. 1F illustrates an input image 6001, and a single iteration of anexpansion operation and a merge operation.

In FIG. 1F the input image 6001 undergoes two expansion operations.

The first expansion operation involves filtering the input image by afirst filtering operation 6031 to provide first regions of interest(denoted 1) in a first filtered image 6031′.

The first expansion operation also involves filtering the input image bya second filtering operation 6032 to provide first regions of interest(denoted 2) in a second filtered image 6032′,

The merge operation includes merging the two images by overlaying thefirst filtered image on the second filtered image to provide regions ofinterest 1, 2, 12 and 21. Region of interest 12 is an overlap areashared by a certain region of interest 1 and a certain region ofinterest 2. Region of interest 21 is a union of another region ofinterest 1 and another region of interest 2.

FIG. 1G illustrates method 5200 for generating a signature.

Method 5200 may include the following sequence of steps:

Step 5210 of receiving or generating an image.

Step 5220 of performing a first iteration expansion operation (which isan expansion operation that is executed during a first iteration)

Step 5230 of performing a first iteration merge operation.

Step 5240 of amending index k (k is an iteration counter). In FIG. 7 inincremented by one—this is only an example of how the number ofiterations are tracked.

Step 5260 of performing a k'th iteration expansion operation on the(k−1)'th iteration merge results.

Step 5270 of performing a k'th iteration merge operation (on the k'thiteration expansion operation results.

Step 5280 of changing the value of index k.

Step 5290 of checking if all iteration ended (k reached its finalvalue—for example K).

If no—there are still iterations to be executed—jumping from step 5290to step 5260.

If yes—jumping to step 5060 of completing the generation of thesignature. This may include, for example, selecting significantattributes, determining retrieval information (for example indexes) thatpoint to the selected significant attributes.

Step 5220 may include:

Step 5222 of generating multiple representations of the image within amulti-dimensional space of f(1) dimensions. The expansion operation ofstep 5220 generates a first iteration multidimensional representation ofthe first image. The number of dimensions of this first iterationmultidimensional representation is denoted f(1).

Step 5224 of assigning a unique index for each region of interest withinthe multiple representations. For example, referring to FIG. 6—indexes 1and indexes 2 are assigned to regions of interests generated during thefirst iteration expansion operations 6031 and 6032.

Step 5230 may include:

Step 5232 of searching for relationships between regions of interest anddefine regions of interest that are related to the relationships. Forexample—union or intersection illustrate din FIG. 6.

Step 5234 of assigning a unique index for each region of interest withinthe multiple representations. For example—referring to FIG. 6—indexes 1,2, 12 and 21.

Step 5260 may include:

Step 5262 of generating multiple representations of the merge results ofthe (k−1)'th iteration within a multi-dimensional space of f(k)dimensions. The expansion operation of step 5260 generates a k'thiteration multidimensional representation of the first image. The numberof dimensions of this kth iteration multidimensional representation isdenoted f(k).

Step 5264 of assigning a unique index for each region of interest withinthe multiple representations.

Step 5270 may include

Step 5272 of searching for relationships between regions of interest anddefine regions of interest that are related to the relationships.

Step 5274 of Assigning a unique index for each region of interest withinthe multiple representations.

FIG. 1H illustrates a method 5201. In method 5201 the relationshipsbetween the regions of interest are overlaps.

Thus—step 5232 is replaced by step 5232′ of searching for overlapsbetween regions of interest and define regions of interest that arerelated to the overlaps.

Step 5272 is replaced by step 5272′ of searching for overlaps betweenregions of interest and define regions of interest that are related tothe overlaps.

FIG. 1I illustrates a method 7000 for low-power calculation of asignature.

Method 7000 starts by step 7010 of receiving or generating a media unitof multiple objects.

Step 7010 may be followed by step 7012 of processing the media unit byperforming multiple iterations, wherein at least some of the multipleiterations comprises applying, by spanning elements of the iteration,dimension expansion process that are followed by a merge operation.

The applying of the dimension expansion process of an iteration mayinclude (a) determining a relevancy of the spanning elements of theiteration; and (b) completing the dimension expansion process byrelevant spanning elements of the iteration and reducing a powerconsumption of irrelevant spanning elements until, at least, acompletion of the applying of the dimension expansion process.

The identifiers may be retrieval information for retrieving thesignificant portions.

The at least some of the multiple iterations may be a majority of themultiple iterations.

The output of the multiple iteration may include multiple propertyattributes for each segment out of multiple segments of the media unit;and wherein the significant portions of an output of the multipleiterations may include more impactful property attributes.

The first iteration of the multiple iteration may include applying thedimension expansion process by applying different filters on the mediaunit.

The at least some of the multiple iteration exclude at least a firstiteration of the multiple iterations. See, for example, FIG. 1E.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration that preceded the iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on properties of the media unit.

The determining the relevancy of the spanning elements of the iterationmay be performed by the spanning elements of the iteration.

Method 7000 may include a neural network processing operation that maybe executed by one or more layers of a neural network and does notbelong to the at least some of the multiple iterations. See, forexample, FIG. 1E.

The at least one iteration may be executed without reducing powerconsumption of irrelevant neurons of the one or more layers.

The one or more layers may output information about properties of themedia unit, wherein the information differs from a recognition of themultiple objects.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative of an identity of the relevantspanning elements of the iteration. See, for example, FIG. 1C.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative a history of dimension expansionprocesses until the iteration that differs from the first iteration.

The each spanning element may be associated with a subset of referenceidentifiers. The determining of the relevancy of each spanning elementsof the iteration may be based a relationship between the subset of thereference identifiers of the spanning element and an output of a lastmerge operation before the iteration.

The output of a dimension expansion process of an iteration may be amultidimensional representation of the media unit that may include mediaunit regions of interest that may be associated with one or moreexpansion processes that generated the regions of interest.

The merge operation of the iteration may include selecting a subgroup ofmedia unit regions of interest based on a spatial relationship betweenthe subgroup of multidimensional regions of interest. See, for example,FIGS. 3 and 6.

Method 7000 may include applying a merge function on the subgroup ofmultidimensional regions of interest. See, for example, FIGS. 1C and 1F.

Method 7000 may include applying an intersection function on thesubgroup of multidimensional regions of interest. See, for example,FIGS. 1C and 1F.

The merge operation of the iteration may be based on an actual size ofone or more multidimensional regions of interest.

The merge operation of the iteration may be based on relationshipbetween sizes of the multidimensional regions of interest. Forexample—larger multidimensional regions of interest may be maintainedwhile smaller multidimensional regions of interest may be ignored of.

The merge operation of the iteration may be based on changes of themedia unit regions of interest during at least the iteration and one ormore previous iteration.

Step 7012 may be followed by step 7014 of determining identifiers thatare associated with significant portions of an output of the multipleiterations.

Step 7014 may be followed by step 7016 of providing a signature thatcomprises the identifiers and represents the multiple objects.

Localization and Segmentation

Any of the mentioned above signature generation method provides asignature that does not explicitly includes accurate shape information.This adds to the robustness of the signature to shape relatedinaccuracies or to other shape related parameters.

The signature includes identifiers for identifying media regions ofinterest.

Each media region of interest may represent an object (for example avehicle, a pedestrian, a road element, a human made structure,wearables, shoes, a natural element such as a tree, the sky, the sun,and the like) or a part of an object (for example—in the case of thepedestrian—a neck, a head, an arm, a leg, a thigh, a hip, a foot, anupper arm, a forearm, a wrist, and a hand). It should be noted that forobject detection purposes a part of an object may be regarded as anobject.

The exact shape of the object may be of interest.

FIG. 1J illustrates method 7002 of generating a hybrid representation ofa media unit.

Method 7002 may include a sequence of steps 7020, 7022, 7024 and 7026.

Step 7020 may include receiving or generating the media unit.

Step 7022 may include processing the media unit by performing multipleiterations, wherein at least some of the multiple iterations comprisesapplying, by spanning elements of the iteration, dimension expansionprocess that are followed by a merge operation.

Step 7024 may include selecting, based on an output of the multipleiterations, media unit regions of interest that contributed to theoutput of the multiple iterations.

Step 7026 may include providing a hybrid representation, wherein thehybrid representation may include (a) shape information regarding shapesof the media unit regions of interest, and (b) a media unit signaturethat includes identifiers that identify the media unit regions ofinterest.

Step 7024 may include selecting the media regions of interest persegment out of multiple segments of the media unit. See, for example,FIG. 2.

Step 7026 may include step 7027 of generating the shape information.

The shape information may include polygons that represent shapes thatsubstantially bound the media unit regions of interest. These polygonsmay be of a high degree.

In order to save storage space, the method may include step 7028 ofcompressing the shape information of the media unit to providecompressed shape information of the media unit.

FIG. 1K illustrates method 5002 for generating a hybrid representationof a media unit.

Method 5002 may start by step 5011 of receiving or generating a mediaunit.

Step 5011 may be followed by processing the media unit by performingmultiple iterations, wherein at least some of the multiple iterationscomprises applying, by spanning elements of the iteration, dimensionexpansion process that are followed by a merge operation.

The processing may be followed by steps 5060 and 5062.

The processing may include steps 5020, 5030, 5040 and 5050.

Step 5020 may include performing a k'th iteration expansion process (kmay be a variable that is used to track the number of iterations).

Step 5030 may include performing a k'th iteration merge process.

Step 5040 may include changing the value of k.

Step 5050 may include checking if all required iterations were done—ifso proceeding to steps 5060 and 5062. Else—jumping to step 5020.

The output of step 5020 is a k'th iteration expansion result.

The output of step 5030 is a k'th iteration merge result.

For each iteration (except the first iteration)—the merge result of theprevious iteration is an input to the current iteration expansionprocess.

Step 5060 may include completing the generation of the signature.

Step 5062 may include generating shape information regarding shapes ofmedia unit regions of interest. The signature and the shape informationprovide a hybrid representation of the media unit.

The combination of steps 5060 and 5062 amounts to a providing a hybridrepresentation, wherein the hybrid representation may include (a) shapeinformation regarding shapes of the media unit regions of interest, and(b) a media unit signature that includes identifiers that identify themedia unit regions of interest.

FIG. 1L illustrates method 5203 for generating a hybrid representationof an image.

Method 5200 may include the following sequence of steps:

Step 5210 of receiving or generating an image.

Step 5230 of performing a first iteration expansion operation (which isan expansion operation that is executed during a first iteration)

Step 5240 of performing a first iteration merge operation.

Step 5240 of amending index k (k is an iteration counter). In FIG. 1L inincremented by one—this is only an example of how the number ofiterations are tracked.

Step 5260 of performing a k'th iteration expansion operation on the(k−1)'th iteration merge results.

Step 5270 of Performing a k'th iteration merge operation (on the k'thiteration expansion operation results.

Step 5280 of changing the value of index k.

Step 5290 of checking if all iteration ended (k reached its finalvalue—for example K).

If no—there are still iterations to be executed—jumping from step 5290to step 5260.

If yes—jumping to step 5060.

Step 5060 may include completing the generation of the signature. Thismay include, for example, selecting significant attributes, determiningretrieval information (for example indexes) that point to the selectedsignificant attributes.

Step 5062 may include generating shape information regarding shapes ofmedia unit regions of interest. The signature and the shape informationprovide a hybrid representation of the media unit.

The combination of steps 5060 and 5062 amounts to a providing a hybridrepresentation, wherein the hybrid representation may include (a) shapeinformation regarding shapes of the media unit regions of interest, and(b) a media unit signature that includes identifiers that identify themedia unit regions of interest.

Step 5220 may include:

Step 5222 of generating multiple representations of the image within amulti-dimensional space of f(k) dimensions.

Step 5224 of assigning a unique index for each region of interest withinthe multiple representations. (for example, referring to FIG. 1F—indexes1 and indexes 2 following first iteration expansion operations 6031 and6032.

Step 5230 may include

Step 5226 of searching for relationships between regions of interest anddefine regions of interest that are related to the relationships. Forexample—union or intersection illustrated in FIG. 1F.

Step 5228 of assigning a unique index for each region of interest withinthe multiple representations. For example—referring to FIG. 1F—indexes1, 2, 12 and 21.

Step 5260 may include:

Step 5262 of generating multiple representations of the merge results ofthe (k−1)'th iteration within a multi-dimensional space of f(k)dimensions. The expansion operation of step 5260 generates a k'thiteration multidimensional representation of the first image. The numberof dimensions of this kth iteration multidimensional representation isdenoted f(k).

Step 5264 of assigning a unique index for each region of interest withinthe multiple representations.

Step 5270 may include

Step 5272 of searching for relationships between regions of interest anddefine regions of interest that are related to the relationships.

Step 5274 of assigning a unique index for each region of interest withinthe multiple representations.

FIG. 1M illustrates method 5205 for generating a hybrid representationof an image.

Method 5200 may include the following sequence of steps:

Step 5210 of receiving or generating an image.

Step 5230 of performing a first iteration expansion operation (which isan expansion operation that is executed during a first iteration)

Step 5240 of performing a first iteration merge operation.

Step 5240 of amending index k (k is an iteration counter). In FIG. 1M inincremented by one—this is only an example of how the number ofiterations are tracked.

Step 5260 of performing a k'th iteration expansion operation on the(k−1)'th iteration merge results.

Step 5270 of performing a k'th iteration merge operation (on the k'thiteration expansion operation results.

Step 5280 of changing the value of index k.

Step 5290 of checking if all iteration ended (k reached its finalvalue—for example K).

If no—there are still iterations to be executed—jumping from step 5290to step 5260.

If yes—jumping to steps 5060 and 5062.

Step 5060 may include completing the generation of the signature. Thismay include, for example, selecting significant attributes, determiningretrieval information (for example indexes) that point to the selectedsignificant attributes.

Step 5062 may include generating shape information regarding shapes ofmedia unit regions of interest. The signature and the shape informationprovide a hybrid representation of the media unit.

The combination of steps 5060 and 5062 amounts to a providing a hybridrepresentation, wherein the hybrid representation may include (a) shapeinformation regarding shapes of the media unit regions of interest, and(b) a media unit signature that includes identifiers that identify themedia unit regions of interest.

Step 5220 may include:

Step 5221 of filtering the image with multiple filters that areorthogonal to each other to provide multiple filtered images that arerepresentations of the image in a multi-dimensional space of f(1)dimensions. The expansion operation of step 5220 generates a firstiteration multidimensional representation of the first image. The numberof filters is denoted f(1).

Step 5224 of assigning a unique index for each region of interest withinthe multiple representations. (for example, referring to FIG. 1F—indexes1 and indexes 2 following first iteration expansion operations 6031 and6032.

Step 5230 may include

Step 5226 of searching for relationships between regions of interest anddefine regions of interest that are related to the relationships. Forexample—union or intersection illustrated in FIG. 1F.

Step 5228 of assigning a unique index for each region of interest withinthe multiple representations. For example—referring to FIG. 1F—indexes1, 2, 12 and 21.

Step 5260 may include:

Step 5262 of generating multiple representations of the merge results ofthe (k−1)'th iteration within a multi-dimensional space of f(k)dimensions. The expansion operation of step 5260 generates a k'thiteration multidimensional representation of the first image. The numberof dimensions of this kth iteration multidimensional representation isdenoted f(k).

Step 5264 of assigning a unique index for each region of interest withinthe multiple representations.

Step 5270 may include

Step 5272 of searching for relationships between regions of interest anddefine regions of interest that are related to the relationships.

Step 5274 of assigning a unique index for each region of interest withinthe multiple representations.

The filters may be orthogonal may be non-orthogonal—for example bedecorrelated. Using non-orthogonal filters may increase the number offilters—but this increment may be tolerable.

Object Detection Using Compressed Shape Information

Object detection may include comparing a signature of an input image tosignatures of one or more cluster structures in order to find one ormore cluster structures that include one or more matching signaturesthat match the signature of the input image.

The number of input images that are compared to the cluster structuresmay well exceed the number of signatures of the cluster structures. Forexample—thousands, tens of thousands, hundreds of thousands (and evenmore) of input signature may be compared to much less cluster structuresignatures. The ratio between the number of input images to theaggregate number of signatures of all the cluster structures may exceedten, one hundred, one thousand, and the like.

In order to save computational resources, the shape information of theinput images may be compressed.

On the other hand—the shape information of signatures that belong to thecluster structures may be uncompressed—and of higher accuracy than thoseof the compressed shape information.

When the higher quality is not required—the shape information of thecluster signature may also be compressed.

Compression of the shape information of cluster signatures may be basedon a priority of the cluster signature, a popularity of matches to thecluster signatures, and the like.

The shape information related to an input image that matches one or moreof the cluster structures may be calculated based on shape informationrelated to matching signatures.

For example—a shape information regarding a certain identifier withinthe signature of the input image may be determined based on shapeinformation related to the certain identifiers within the matchingsignatures.

Any operation on the shape information related to the certainidentifiers within the matching signatures may be applied in order todetermine the (higher accuracy) shape information of a region ofinterest of the input image identified by the certain identifier.

For example—the shapes may be virtually overlaid on each other and thepopulation per pixel may define the shape.

For example—only pixels that appear in at least a majority of theoverlaid shaped should be regarded as belonging to the region ofinterest.

Other operations may include smoothing the overlaid shapes, selectingpixels that appear in all overlaid shapes.

The compressed shape information may be ignored of or be taken intoaccount.

FIG. 1N illustrates method 7003 of determining shape information of aregion of interest of a media unit.

Method 7003 may include a sequence of steps 7030, 7032 and 7034.

Step 7030 may include receiving or generating a hybrid representation ofa media unit. The hybrid representation includes compressed shapeinformation.

Step 7032 may include comparing the media unit signature of the mediaunit to signatures of multiple concept structures to find a matchingconcept structure that has at least one matching signature that matchesto the media unit signature.

Step 7034 may include calculating higher accuracy shape information thatis related to regions of interest of the media unit, wherein the higheraccuracy shape information is of higher accuracy than the compressedshape information of the media unit, wherein the calculating is based onshape information associated with at least some of the matchingsignatures.

Step 7034 may include at least one out of:

Determining shapes of the media unit regions of interest using thehigher accuracy shape information.

For each media unit region of interest, virtually overlaying shapes ofcorresponding media units of interest of at least some of the matchingsignatures.

FIG. 1O illustrates a matching process and a generation of a higheraccuracy shape information.

It is assumed that there are multiple (M) cluster structures4974(1)-4974(M). Each cluster structure includes cluster signatures,metadata regarding the cluster signatures, and shape informationregarding the regions of interest identified by identifiers of thecluster signatures.

For example—first cluster structure 4974(1) includes multiple (N1)signatures (referred to as cluster signatures CS) CS(1,1)-CS(1,N1)4975(1,1)-4975(1,N1), metadata 4976(1), and shape information (Shapeinfo4977(1)) regarding shapes of regions of interest associated withidentifiers of the CSs.

Yet for another example—M'th cluster structure 4974(M) includes multiple(N2) signatures (referred to as cluster signatures CS) CS(M,1)-CS(M,N2)4975(M,1)-4975(M,N2), metadata 4976(M), and shape information (Shapeinfo4977(M)) regarding shapes of regions of interest associated withidentifiers of the CSs.

The number of signatures per concept structure may change over time—forexample due to cluster reduction attempts during which a CS is removedfrom the structure to provide a reduced cluster structure, the reducedstructure structure is checked to determine that the reduced clustersignature may still identify objects that were associated with the(non-reduced) cluster signature—and if so the signature may be reducedfrom the cluster signature.

The signatures of each cluster structures are associated to each other,wherein the association may be based on similarity of signatures and/orbased on association between metadata of the signatures.

Assuming that each cluster structure is associated with a uniqueobject—then objects of a media unit may be identified by finding clusterstructures that are associated with said objects. The finding of thematching cluster structures may include comparing a signature of themedia unit to signatures of the cluster structures- and searching forone or more matching signature out of the cluster signatures.

In FIG. 1O—a media unit having a hybrid representation undergoes objectdetection. The hybrid representation includes media unit signature 4972and compressed shape information 4973.

The media unit signature 4972 is compared to the signatures of the Mcluster structures—from CS(1,1) 4975(1,1) till CS(M,N2) 4975(M,N2).

We assume that one or more cluster structures are matching clusterstructures.

Once the matching cluster structures are found the method proceeds bygenerating shape information that is of higher accuracy then thecompressed shape information.

The generation of the shape information is done per identifier.

For each j that ranges between 1 and J (J is the number of identifiersper the media unit signature 4972) the method may perform the steps of:

Find (step 4978(j)) the shape information of the j'th identifier of eachmatching signature- or of each signature of the matching clusterstructure.

Generate (step 4979(j)) a higher accuracy shape information of the j'thidentifier.

For example—assuming that the matching signatures include CS(1,1)2975(1,1), CS(2,5) 2975(2,5), CS(7,3) 2975(7,3) and CS(15,2) 2975(15,2),and that the j'th identifier is included in CS(1,1) 2975(1,1), CS(7,3)2975(7,3) and CS(15,2) 2975(15,2)—then the shape information of the j'thidentifier of the media unit is determined based on the shapeinformation associated with CS(1,1) 2975(1,1),CS(7,3) 2975(7,3) andCS(15,2) 2975(15,2).

FIG. 1P illustrates an image 8000 that includes four regions of interest8001, 8002, 8003 and 8004. The signature 8010 of image 8000 includesvarious identifiers including ID1 8011, ID2 8012, ID3 8013 and ID4 8014that identify the four regions of interest 8001, 8002, 8003 and 8004.

The shapes of the four regions of interest 8001, 8002, 8003 and 8004 arefour polygons. Accurate shape information regarding the shapes of theseregions of interest may be generated during the generation of signature8010.

FIG. 1Q illustrates the compressing of the shape information torepresent a compressed shape information that reflects simplerapproximations (8001′, 8002′, 8003′ and 8004′) of the regions ofinterest 8001, 8002, 8003 and 8004. In this example simpler may includeless facets, fewer values of angles, and the like.

The hybrid representation of the media unit, after compression representan media unit with simplified regions of interest 8001′, 8002′, 8003′and 8004′-as shown in FIG. 1R.

Scale Based Bootstrap

Objects may appear in an image at different scales. Scale invariantobject detection may improve the reliability and repeatability of theobject detection and may also use fewer number of clusterstructures—thus reduced memory resources and also lower computationalresources required to maintain fewer cluster structures.

FIG. 1S illustrates method 8020 for scale invariant object detection.

Method 8020 may include a first sequence of steps that may include step8022, 8024, 8026 and 8028.

Step 8022 may include receiving or generating a first image in which anobject appears in a first scale and a second image in which the objectappears in a second scale that differs from the first scale.

Step 8024 may include generating a first image signature and a secondimage signature.

The first image signature includes a first group of at least one certainfirst image identifier that identifies at least a part of the object.See, for example image 8000′ of FIG. 2A. The person is identified byidentifiers ID6 8016 and ID8 8018 that represent regions of interest8006 and 8008.

The second image signature includes a second group of certain secondimage identifiers that identify different parts of the object.

See, for example image 8000 of FIG. 19. The person is identified byidentifiers ID1 8011, ID2 8012, ID3 8013, and ID4 8014 that representregions of interest 8001, 8002, 8003 and 8004.

The second group is larger than first group—as the second group has moremembers than the first group.

Step 8026 may include linking between the at least one certain firstimage identifier and the certain second image identifiers.

Step 8026 may include linking between the first image signature, thesecond image signature and the object.

Step 8026 may include adding the first signature and the secondsignature to a certain concept structure that is associated with theobject. For example, referring to FIG. 1O, the signatures of the firstand second images may be included in a cluster concept out of4974(1)-4974(M).

Step 8028 may include determining whether an input image includes theobject based, at least in part, on the linking. The input image differsfrom the first and second images.

The determining may include determining that the input image includesthe object when a signature of the input image includes the at least onecertain first image identifier or the certain second image identifiers.

The determining may include determining that the input image includesthe object when the signature of the input image includes only a part ofthe at least one certain first image identifier or only a part of thecertain second image identifiers.

The linking may be performed for more than two images in which theobject appears in more than two scales.

For example, see FIG. 2B in which a person appears at three differentscales—at three different images.

In first image 8051 the person is included in a single region ofinterest 8061 and the signature 8051′ of first image 8051 includes anidentifier ID61 that identifies the single region of interest—identifiesthe person.

In second image 8052 the upper part of the person is included in regionof interest 8068, the lower part of the person is included in region ofinterest 8069 and the signature 8052′ of second image 8052 includesidentifiers ID68 and ID69 that identify regions of interest 8068 and8069 respectively.

In third image 8053 the eyes of the person are included in region ofinterest 8062, the mouth of the person is included in region of interest8063, the head of the person appears in region of interest 8064, theneck and arms of the person appear in region of interest 8065, themiddle part of the person appears in region of interest 8066, and thelower part of the person appears in region of interest 8067. Signature8053′ of third image 8053 includes identifiers ID62, ID63, ID64, ID65,ID55 and ID67 that identify regions of interest 8062-8067 respectively.

Method 8020 may link signatures 8051′, 8052′ and 8053′ to each other.For example—these signatures may be included in the same clusterstructure.

Method 8020 may link (i) ID61, (ii) signatures ID68 and ID69, and (ii)signature ID62, ID63, ID64, ID65, ID66 and ID67.

FIG. 2C illustrates method 8030 for object detection.

Method 8030 may include the steps of method 8020 or may be preceded bysteps 8022, 8024 and 8026.

Method 8030 may include a sequence of steps 8032, 8034, 8036 and 8038.

Step 8032 may include receiving or generating an input image.

Step 8034 may include generating a signature of the input image.

Step 8036 may include comparing the signature of the input image tosignatures of a certain concept structure. The certain concept structuremay be generated by method 8020.

Step 8038 may include determining that the input image comprises theobject when at least one of the signatures of the certain conceptstructure matches the signature of the input image.

FIG. 2D illustrates method 8040 for object detection.

Method 8040 may include the steps of method 8020 or may be preceded bysteps 8022, 8024 and 8026.

Method 8040 may include a sequence of steps 8041, 8043, 8045, 8047 and8049.

Step 8041 may include receiving or generating an input image.

Step 8043 may include generating a signature of the input image, thesignature of the input image comprises only some of the certain secondimage identifiers; wherein the input image of the second scale.

Step 8045 may include changing a scale of the input image to the firstscale to a provide an amended input image.

Step 8047 may include generating a signature of the amended input image.

Step 8049 may include verifying that the input image comprises theobject when the signature of the amended input image comprises the atleast one certain first image identifier.

FIG. 2E illustrates method 8050 for object detection.

Method 8050 may include the steps of method 8020 or may be preceded bysteps 8022, 8024 and 8026.

Method 8050 may include a sequence of steps 8052, 8054, 8056 and 8058.

Step 8052 may include receiving or generating an input image.

Step 8054 may include generating a signature of the input image.

Step 8056 may include searching in the signature of the input image forat least one of (a) the at least one certain first image identifier, and(b) the certain second image identifiers.

Step 8058 may include determining that the input image comprises theobject when the signature of the input image comprises the at least oneof (a) the at least one certain first image identifier, and (b) thecertain second image identifiers.

It should be noted that step 8056 may include searching in the signatureof the input image for at least one of (a) one or more certain firstimage identifier of the at least one certain first image identifier, and(b) at least one certain second image identifier of the certain secondimage identifiers.

It should be noted that step 8058 may include determining that the inputimage includes the object when the signature of the input imagecomprises the at least one of (a) one or more certain first imageidentifier of the at least one certain first image identifier, and (b)the at least one certain second image identifier.

Movement Based Bootstrapping

A single object may include multiple parts that are identified bydifferent identifiers of a signature of the image. In cases such asunsupervised learning, it may be beneficial to link the multiple objectparts to each other without receiving prior knowledge regarding theirinclusion in the object.

Additionally or alternatively, the linking can be done in order toverify a previous linking between the multiple object parts.

FIG. 2F illustrates method 8070 for object detection.

Method 8070 is for movement based object detection.

Method 8070 may include a sequence of steps 8071, 8073, 8075, 8077, 8078and 8079.

Step 8071 may include receiving or generating a video stream thatincludes a sequence of images.

Step 8073 may include generating image signatures of the images. Eachimage is associated with an image signature that comprises identifiers.Each identifier identifiers a region of interest within the image.

Step 8075 may include generating movement information indicative ofmovements of the regions of interest within consecutive images of thesequence of images. Step 8075 may be preceded by or may includegenerating or receiving location information indicative of a location ofeach region of interest within each image. The generating of themovement information is based on the location information.

Step 8077 may include searching, based on the movement information, fora first group of regions of interest that follow a first movement.Different first regions of interest are associated with different partsof an object.

Step 8078 may include linking between first identifiers that identifythe first group of regions of interest.

Step 8079 may include linking between first image signatures thatinclude the first linked identifiers.

Step 8079 may include adding the first image signatures to a firstconcept structure, the first concept structure is associated with thefirst image.

Step 8079 may be followed by determining whether an input image includesthe object based, at least in part, on the linking

An example of various steps of method 8070 is illustrated in FIG. 2H.

FIG. 2G illustrates three images 8091, 8092 and 8093 that were taken atdifferent points in time.

First image 8091 illustrates a gate 8089′ that is located in region ofinterest 8089 and a person that faces the gate. Various parts of theperson are located within regions of interest 8081, 8082, 8083, 8084 and8085.

The first image signature 8091′ includes identifiers ID81, ID82, ID83,ID84, ID85 and ID89 that identify regions of interest 8081, 8082, 8083,8084, 8085 and 8089 respectively.

The first image location information 8091″ includes the locations L81,L82, L83, L84, L85 and L89 of regions of interest 8081, 8082, 8083,8084, 8085 and 8089 respectively. A location of a region of interest mayinclude a location of the center of the region of interest, the locationof the borders of the region of interest or any location informationthat may define the location of the region of interest or a part of theregion of interest.

Second image 8092 illustrates a gate that is located in region ofinterest 8089 and a person that faces the gate. Various parts of theperson are located within regions of interest 8081, 8082, 8083, 8084 and8085. Second image also includes a pole that is located within region ofinterest 8086. In the first image that the pole was concealed by theperson.

The second image signature 8092′ includes identifiers ID81, ID82, ID83,ID84, ID85, ID86, and ID89 that identify regions of interest 8081, 8082,8083, 8084, 8085, 8086 and 8089 respectively.

The second image location information 8092″ includes the locations L81,L82, L83, L84, L85, L86 and L89 of regions of interest 8081, 8082, 8083,8084, 8085, 8086 and 8089 respectively.

Third image 8093 illustrates a gate that is located in region ofinterest 8089 and a person that faces the gate. Various parts of theperson are located within regions of interest 8081, 8082, 8083, 8084 and8085. Third image also includes a pole that is located within region ofinterest 8086, and a balloon that is located within region of interest8087.

The third image signature 8093′ includes identifiers ID81, ID82, ID83,ID84, ID85, ID86, ID87 and ID89 that identify regions of interest 8081,8082, 8083, 8084, 8085, 8086, 8087 and 8089 respectively.

The third image location information 8093″ includes the locations L81,L82, L83, L84, L85, L86, L87 and L89 of regions of interest 8081, 8082,8083, 8084, 8085, 8086, 8086 and 8089 respectively.

The motion of the various regions of interest may be calculated bycomparing the location information related to different images. Themovement information may take into account the different in theacquisition time of the images.

The comparison shows that regions of interest 8081, 8082, 8083, 8084,8085 move together and thus they should be linked to each other—and itmay be assumed that they all belong to the same object.

FIG. 2H illustrates method 8100 for object detection.

Method 8100 may include the steps of method 8070 or may be preceded bysteps 8071, 8073, 8075, 8077 and 8078.

Method 8100 may include the following sequence of steps:

Step 8102 of receiving or generating an input image.

Step 8104 of generating a signature of the input image.

Step 8106 of comparing the signature of the input image to signatures ofa first concept structure. The first concept structure includes firstidentifiers that were linked to each other based on movements of firstregions of interest that are identified by the first identifiers.

Step 8108 of determining that the input image includes a first objectwhen at least one of the signatures of the first concept structurematches the signature of the input image.

FIG. 2I illustrates method 8110 for object detection.

Method 8110 may include the steps of method 8070 or may be preceded bysteps 8071, 8073, 8075, 8077 and 8078.

Method 8110 may include the following sequence of steps:

Step 8112 of receiving or generating an input image.

Step 8114 of generating a signature of the input image.

Step 8116 of searching in the signature of the input image for at leastone of the first identifiers.

Step 8118 of determining that the input image comprises the object whenthe signature of the input image comprises at least one of the firstidentifiers.

Object Detection that is Robust to Angle of Acquisition

Object detection may benefit from being robust to the angle ofacquisition—to the angle between the optical axis of an image sensor anda certain part of the object. This allows the detection process to bemore reliable, use fewer different clusters (may not require multipleclusters for identifying the same object from different images).

FIG. 2J illustrates method 8120 that includes the following steps:

Step 8122 of receiving or generating images of objects taken fromdifferent angles.

Step 8124 of finding images of objects taken from different angles thatare close to each other. Close enough may be less than 1, 5, 10, 15 and20 degrees—but the closeness may be better reflected by the reception ofsubstantially the same signature.

Step 8126 of linking between the images of similar signatures. This mayinclude searching for local similarities. The similarities are local inthe sense that they are calculated per a subset of signatures. Forexample—assuming that the similarity is determined per two images—then afirst signature may be linked to a second signature that is similar tothe first image. A third signature may be linked to the second imagebased on the similarity between the second and third signatures- andeven regardless of the relationship between the first and thirdsignatures.

Step 8126 may include generating a concept data structure that includesthe similar signatures.

This so-called local or sliding window approach, in addition to theacquisition of enough images (that will statistically provide a largeangular coverage) will enable to generate a concept structure thatinclude signatures of an object taken at multiple directions.

FIG. 2K illustrates a person 8130 that is imaged from different angles(8131, 8132, 8133, 8134, 8135 and 8136). While the signature of a frontview of the person (obtained from angle 8131) differs from the signatureof the side view of the person (obtained from angle 8136), the signatureof images taken from multiple angles between angles 8141 and 8136compensates for the difference—as the difference between images obtainedfrom close angles are similar (local similarity) to each other.

Signature Tailored Matching Threshold

Object detection may be implemented by (a) receiving or generatingconcept structures that include signatures of media units and relatedmetadata, (b) receiving a new media unit, generating a new media unitsignature, and (c) comparing the new media unit signature to the conceptsignatures of the concept structures.

The comparison may include comparing new media unit signatureidentifiers (identifiers of objects that appear in the new media unit)to concept signature identifiers and determining, based on a signaturematching criteria whether the new media unit signature matches a conceptsignature. If such a match is found then the new media unit is regardedas including the object associated with that concept structure.

It was found that by applying an adjustable signature matching criteria,the matching process may be highly effective and may adapt itself to thestatistics of appearance of identifiers in different scenarios. Forexample—a match may be obtained when a relatively rear but highlydistinguishing identifier appears in the new media unit signature and ina cluster signature, but a mismatch may be declared when multiple commonand slightly distinguishing identifiers appear in the new media unitsignature and in a cluster signature.

FIG. 2L illustrates method 8200 for object detection.

Method 8200 may include:

Step 8210 of receiving an input image.

Step 8212 of generating a signature of the input image.

Step 8214 of comparing the signature of the input image to signatures ofa concept structure.

Step 8216 of determining whether the signature of the input imagematches any of the signatures of the concept structure based onsignature matching criteria, wherein each signature of the conceptstructure is associated within a signature matching criterion that isdetermined based on an object detection parameter of the signature.

Step 8218 of concluding that the input image comprises an objectassociated with the concept structure based on an outcome of thedetermining.

The signature matching criteria may be a minimal number of matchingidentifiers that indicate of a match. For example—assuming a signaturethat include few tens of identifiers, the minimal number may varybetween a single identifier to all of the identifiers of the signature.

It should be noted that an input image may include multiple objects andthat an signature of the input image may match multiple clusterstructures. Method 8200 is applicable to all of the matching processes-and that the signature matching criteria may be set for each signatureof each cluster structure.

Step 8210 may be preceded by step 8202 of determining each signaturematching criterion by evaluating object detection capabilities of thesignature under different signature matching criteria.

Step 8202 may include:

Step 8203 of receiving or generating signatures of a group of testimages.

Step 8204 of calculating the object detection capability of thesignature, for each signature matching criterion of the differentsignature matching criteria.

Step 8206 of selecting the signature matching criterion based on theobject detection capabilities of the signature under the differentsignature matching criteria.

The object detection capability may reflect a percent of signatures ofthe group of test images that match the signature.

The selecting of the signature matching criterion comprises selectingthe signature matching criterion that once applied results in a percentof signatures of the group of test images that match the signature thatis closets to a predefined desired percent of signatures of the group oftest images that match the signature.

The object detection capability may reflect a significant change in thepercent of signatures of the group of test images that match thesignature. For example—assuming, that the signature matching criteria isa minimal number of matching identifiers and that changing the value ofthe minimal numbers may change the percentage of matching test images. Asubstantial change in the percentage (for example a change of more than10, 20, 30, 40 percent) may be indicative of the desired value. Thedesired value may be set before the substantial change, proximate to thesubstantial change, and the like.

For example, referring to FIG. 1O, cluster signatures CS(1,1), CS(2,5),CS(7,3) and CS(15,2) match unit signature 4972. Each of these matchesmay apply a unique signature matching criterion.

FIG. 2M illustrates method 8220 for object detection.

Method 8220 is for managing a concept structure.

Method 8220 may include:

Step 8222 of determining to add a new signature to the conceptstructure. The concept structure may already include at least one oldsignature. The new signature includes identifiers that identify at leastparts of objects.

Step 8224 of determining a new signature matching criterion that isbased on one or more of the identifiers of the new signature. The newsignature matching criterion determines when another signature matchesthe new signature. The determining of the new signature matchingcriterion may include evaluating object detection capabilities of thesignature under different signature matching criteria.

Step 8224 may include steps 8203, 8204 and 8206 (include din step 8206)of method 8200.

Examples of Systems

FIG. 22N illustrates an example of a system capable of executing one ormore of the mentioned above methods.

The system include various components, elements and/or units.

A component element and/or unit may be a processing circuitry may beimplemented as a central processing unit (CPU), and/or one or more otherintegrated circuits such as application-specific integrated circuits(ASICs), field programmable gate arrays (FPGAs), full-custom integratedcircuits, etc., or a combination of such integrated circuits.

Alternatively, each component element and/or unit may implemented inhardware, firmware, or software that may be executed by a processingcircuitry.

System 4900 may include sensing unit 4902, communication unit 4904,input 4911, processor 4950, and output 4919. The communication unit 4904may include the input and/or the output.

Input and/or output may be any suitable communications component such asa network interface card, universal serial bus (USB) port, disk reader,modem or transceiver that may be operative to use protocols such as areknown in the art to communicate either directly, or indirectly, withother elements of the system.

Processor 4950 may include at least some out of

Multiple spanning elements 4951(q).

Multiple merge elements 4952(r).

Object detector 4953.

Cluster manager 4954.

Controller 4955.

Selection unit 4956.

Object detection determination unit 4957.

Signature generator 4958.

Movement information unit 4959.

Identifier unit 4960.

There may be provided a method for low-power calculation of a signature,the method may include receiving or generating a media unit of multipleobjects; processing the media unit by performing multiple iterations,wherein at least some of the multiple iterations may include applying,by spanning elements of the iteration, dimension expansion process thatmay be followed by a merge operation; wherein the applying of thedimension expansion process of an iteration may include determining arelevancy of the spanning elements of the iteration; completing thedimension expansion process by relevant spanning elements of theiteration and reducing a power consumption of irrelevant spanningelements until, at least, a completion of the applying of the dimensionexpansion process; determining identifiers that may be associated withsignificant portions of an output of the multiple iterations; andproviding a signature that may include the identifiers and representsthe multiple objects.

The identifiers may be retrieval information for retrieving thesignificant portions.

The at least some of the multiple iterations may be a majority of themultiple iterations.

The output of the multiple iteration may include multiple propertyattributes for each segment out of multiple segments of the media unit;and wherein the significant portions of an output of the multipleiterations may include more impactful property attributes.

A first iteration of the multiple iteration may include applying thedimension expansion process by applying different filters on the mediaunit.

The at least some of the multiple iteration exclude at least a firstiteration of the multiple iterations.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration that preceded the iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on properties of the media unit.

The determining the relevancy of the spanning elements of the iterationmay be performed by the spanning elements of the iteration.

The method may include a neural network processing operation that may beexecuted by one or more layers of a neural network and does not belongto the at least some of the multiple iterations.

The at least one iteration may be executed without reducing powerconsumption of irrelevant neurons of the one or more layers.

The one or more layers output information about properties of the mediaunit, wherein the information differs from a recognition of the multipleobjects.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative of an identity of the relevantspanning elements of the iteration.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative a history of dimension expansionprocesses until the iteration that differs from the first iteration.

Each spanning element may be associated with a subset of referenceidentifiers; and wherein the determining of the relevancy of eachspanning elements of the iteration may be based a relationship betweenthe subset of the reference identifiers of the spanning element and anoutput of a last merge operation before the iteration.

An output of a dimension expansion process of an iteration may be amultidimensional representation of the media unit that may include mediaunit regions of interest that may be associated with one or moreexpansion processes that generated the regions of interest.

A merge operation of the iteration may include selecting a subgroup ofmedia unit regions of interest based on a spatial relationship betweenthe subgroup of multidimensional regions of interest.

The method may include applying a merge function on the subgroup ofmultidimensional regions of interest.

The method may include applying an intersection function on the subgroupof multidimensional regions of interest.

A merge operation of the iteration may be based on an actual size of oneor more multidimensional regions of interest.

A merge operation of the iteration may be based on relationship betweensizes of the multidimensional regions of interest.

A merge operation of the iteration may be based on changes of the mediaunit regions of interest during at least the iteration and one or moreprevious iteration.

There may be provided a non-transitory computer readable medium forlow-power calculation of a signature, the non-transitory computerreadable medium may store instructions for receiving or generating amedia unit of multiple objects; processing the media unit by performingmultiple iterations, wherein at least some of the multiple iterationsmay include applying, by spanning elements of the iteration, dimensionexpansion process that may be followed by a merge operation; wherein theapplying of the dimension expansion process of an iteration may includedetermining a relevancy of the spanning elements of the iteration;completing the dimension expansion process by relevant spanning elementsof the iteration and reducing a power consumption of irrelevant spanningelements until, at least, a completion of the applying of the dimensionexpansion process; determining identifiers that may be associated withsignificant portions of an output of the multiple iterations; andproviding a signature that may include the identifiers and representsthe multiple objects.

The identifiers may be retrieval information for retrieving thesignificant portions.

The at least some of the multiple iterations may be a majority of themultiple iterations.

The output of the multiple iteration may include multiple propertyattributes for each segment out of multiple segments of the media unit;and wherein the significant portions of an output of the multipleiterations may include more impactful property attributes.

A first iteration of the multiple iteration may include applying thedimension expansion process by applying different filters on the mediaunit.

The at least some of the multiple iteration exclude at least a firstiteration of the multiple iterations.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration that preceded the iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on properties of the media unit.

The determining the relevancy of the spanning elements of the iterationmay be performed by the spanning elements of the iteration.

The non-transitory computer readable medium may store instructions forperforming a neural network processing operation, by one or more layersof a neural network, wherein the neural network processing operation anddoes not belong to the at least some of the multiple iterations.

The non-transitory computer readable medium the at least one iterationmay be executed without reducing power consumption of irrelevant neuronsof the one or more layers.

The one or more layers output information about properties of the mediaunit, wherein the information differs from a recognition of the multipleobjects.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative of an identity of the relevantspanning elements of the iteration.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative a history of dimension expansionprocesses until the iteration that differs from the first iteration.

Each spanning element may be associated with a subset of referenceidentifiers; and wherein the determining of the relevancy of eachspanning elements of the iteration may be based a relationship betweenthe subset of the reference identifiers of the spanning element and anoutput of a last merge operation before the iteration.

An output of a dimension expansion process of an iteration may be amultidimensional representation of the media unit that may include mediaunit regions of interest that may be associated with one or moreexpansion processes that generated the regions of interest.

A merge operation of the iteration may include selecting a subgroup ofmedia unit regions of interest based on a spatial relationship betweenthe subgroup of multidimensional regions of interest.

The non-transitory computer readable medium may store instructions forapplying a merge function on the subgroup of multidimensional regions ofinterest.

The non-transitory computer readable medium may store instructions forapplying an intersection function on the subgroup of multidimensionalregions of interest.

A merge operation of the iteration may be based on an actual size of oneor more multidimensional regions of interest.

A merge operation of the iteration may be based on relationship betweensizes of the multidimensional regions of interest.

A merge operation of the iteration may be based on changes of the mediaunit regions of interest during at least the iteration and one or moreprevious iteration.

There may be provided a signature generator that may include an inputthat may be configured to receive or generate a media unit of multipleobjects; an processor that may be configured to process the media unitby performing multiple iterations, wherein at least some of the multipleiterations may include applying, by spanning elements of the iteration,dimension expansion process that may be followed by a merge operation;wherein the applying of the dimension expansion process of an iterationmay include determining a relevancy of the spanning elements of theiteration; completing the dimension expansion process by relevantspanning elements of the iteration and reducing a power consumption ofirrelevant spanning elements until, at least, a completion of theapplying of the dimension expansion process; an identifier unit that maybe configured to determine identifiers that may be associated withsignificant portions of an output of the multiple iterations; and anoutput that may be configured to provide a signature that may includethe identifiers and represents the multiple objects.

The identifiers may be retrieval information for retrieving thesignificant portions.

The at least some of the multiple iterations may be a majority of themultiple iterations.

The output of the multiple iteration may include multiple propertyattributes for each segment out of multiple segments of the media unit;and wherein the significant portions of an output of the multipleiterations may include more impactful property attributes.

A first iteration of the multiple iteration may include applying thedimension expansion process by applying different filters on the mediaunit.

The at least some of the multiple iteration exclude at least a firstiteration of the multiple iterations.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on at least some identities of relevant spanning elementsof at least one previous iteration that preceded the iteration.

The determining the relevancy of the spanning elements of the iterationmay be based on properties of the media unit.

The determining the relevancy of the spanning elements of the iterationmay be performed by the spanning elements of the iteration.

The signature generator that may include one or more layers of a neuralnetwork that may be configured to perform a neural network processingoperation, wherein the neural network processing operation does notbelong to the at least some of the multiple iterations.

At least one iteration may be executed without reducing powerconsumption of irrelevant neurons of the one or more layers.

The one or more layers output information about properties of the mediaunit, wherein the information differs from a recognition of the multipleobjects.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative of an identity of the relevantspanning elements of the iteration.

The applying, by spanning elements of an iteration that differs from thefirst iteration, the dimension expansion process may include assigningoutput values that may be indicative a history of dimension expansionprocesses until the iteration that differs from the first iteration.

Each spanning element may be associated with a subset of referenceidentifiers; and wherein the determining of the relevancy of eachspanning elements of the iteration may be based a relationship betweenthe subset of the reference identifiers of the spanning element and anoutput of a last merge operation before the iteration.

An output of a dimension expansion process of an iteration may be amultidimensional representation of the media unit that may include mediaunit regions of interest that may be associated with one or moreexpansion processes that generated the regions of interest.

A merge operation of the iteration may include selecting a subgroup ofmedia unit regions of interest based on a spatial relationship betweenthe subgroup of multidimensional regions of interest.

The signature generator that may be configured to apply a merge functionon the subgroup of multidimensional regions of interest.

The signature generator that may be configured to apply an intersectionfunction on the subgroup of multidimensional regions of interest.

A merge operation of the iteration may be based on an actual size of oneor more multidimensional regions of interest.

A merge operation of the iteration may be based on relationship betweensizes of the multidimensional regions of interest.

A merge operation of the iteration may be based on changes of the mediaunit regions of interest during at least the iteration and one or moreprevious iteration.

There may be provided There may be provided a method for low-powercalculation of a signature of a media unit by a group of calculatingelements, the method may include calculating, multiple attributes ofsegments of the media unit; wherein the calculating may includedetermining, by each calculating element of multiple calculatingelements, a relevancy of the calculation unit to the media unit toprovide irrelevant calculating elements and relevant calculatingelements; reducing a power consumption of each irrelevant calculatingelement; and completing the calculating of the multiple attributes ofsegments of the media unit by the relevant calculating elements;determining identifiers that may be associated with significantattributes out of the multiple attributes of segments of the media unit;and providing a signature that may include the identifiers andrepresents the multiple objects.

The calculating elements may be spanning elements.

Each calculating element may be associated with a subset of one or morereference identifiers; and wherein the determining of a relevancy of thecalculation unit to the media unit may be based on a relationshipbetween the subset and the identifiers related to the media unit.

Each calculating element may be associated with a subset of one or morereference identifiers; and wherein a calculation element may be relevantto the media unit when the identifiers related to the media unit mayinclude each reference identifier of the subset.

The calculating of the multiple attributes of segments of the media unitmay be executed in multiple iterations; and wherein each iteration maybe executed by calculation elements associated with the iteration;wherein the determining, by each calculating element of multiplecalculating elements, of the relevancy of the calculation unit to themedia unit may be executed per iteration.

The multiple iterations may be preceded by a calculation of initialmedia unit attributes by one or more layers of a neural network.

There may be provided a non-transitory computer readable medium forlow-power calculation of a signature of a media unit by a group ofcalculating elements, the non-transitory computer readable medium maystore instructions for calculating, multiple attributes of segments ofthe media unit; wherein the calculating may include determining, by eachcalculating element of multiple calculating elements, a relevancy of thecalculation unit to the media unit to provide irrelevant calculatingelements and relevant calculating elements; reducing a power consumptionof each irrelevant calculating element; and completing the calculatingof the multiple attributes of segments of the media unit by the relevantcalculating elements; determining identifiers that may be associatedwith significant attributes out of the multiple attributes of segmentsof the media unit; and providing a signature that may include theidentifiers and represents the multiple objects.

The calculating elements may be spanning elements.

Each calculating element may be associated with a subset of one or morereference identifiers; and wherein the determining of a relevancy of thecalculation unit to the media unit may be based on a relationshipbetween the subset and the identifiers related to the media unit.

Each calculating element may be associated with a subset of one or morereference identifiers; and wherein a calculation element may be relevantto the media unit when the identifiers related to the media unit mayinclude each reference identifier of the subset.

The calculating of the multiple attributes of segments of the media unitmay be executed in multiple iterations; and wherein each iteration maybe executed by calculation elements associated with the iteration;wherein the determining, by each calculating element of multiplecalculating elements, of the relevancy of the calculation unit to themedia unit may be executed per iteration.

The multiple iterations may be preceded by a calculation of initialmedia unit attributes by one or more layers of a neural network.

There may be provided a signature generator that may include a processorthat may be configured to calculate multiple attributes of segments ofthe media unit; wherein the calculating may include determining, by eachcalculating element of multiple calculating elements of the processor, arelevancy of the calculation unit to the media unit to provideirrelevant calculating elements and relevant calculating elements;reducing a power consumption of each irrelevant calculating element; andcompleting the calculating of the multiple attributes of segments of themedia unit by the relevant calculating elements; an identifier unit thatmay be configured to determine identifiers that may be associated withsignificant attributes out of the multiple attributes of segments of themedia unit; and an output that may be configured to provide a signaturethat may include the identifiers and represents the multiple objects.

The calculating elements may be spanning elements.

Each calculating element may be associated with a subset of one or morereference identifiers; and wherein the determining of a relevancy of thecalculation unit to the media unit may be based on a relationshipbetween the subset and the identifiers related to the media unit.

Each calculating element may be associated with a subset of one or morereference identifiers; and wherein a calculation element may be relevantto the media unit when the identifiers related to the media unit mayinclude each reference identifier of the subset.

The calculating of the multiple attributes of segments of the media unitmay be executed in multiple iterations; and wherein each iteration maybe executed by calculation elements associated with the iteration;wherein the determining, by each calculating element of multiplecalculating elements, of the relevancy of the calculation unit to themedia unit may be executed per iteration.

The multiple iterations may be preceded by a calculation of initialmedia unit attributes by one or more layers of a neural network.

There may be provided a method for generating a hybrid representation ofa media unit, the method may include receiving or generating the mediaunit; processing the media unit by performing multiple iterations,wherein at least some of the multiple iterations may include applying,by spanning elements of the iteration, dimension expansion process thatmay be followed by a merge operation; selecting, based on an output ofthe multiple iterations, media unit regions of interest that contributedto the output of the multiple iterations; and providing the hybridrepresentation, wherein the hybrid representation may include shapeinformation regarding shapes of the media unit regions of interest, anda media unit signature that may include identifiers that identify themedia unit regions of interest.

The selecting of the media regions of interest may be executed persegment out of multiple segments of the media unit.

The shape information may include polygons that represent shapes thatsubstantially bound the media unit regions of interest.

The providing of the hybrid representation of the media unit may includecompressing the shape information of the media unit to providecompressed shape information of the media unit.

The method may include comparing the media unit signature of the mediaunit to signatures of multiple concept structures to find a matchingconcept structure that has at least one matching signature that matchesto the media unit signature; and calculating higher accuracy shapeinformation that may be related to regions of interest of the mediaunit, wherein the higher accuracy shape information may be of higheraccuracy than the compressed shape information of the media unit,wherein the calculating may be based on shape information associatedwith at least some of the matching signatures.

The method may include determining shapes of the media unit regions ofinterest using the higher accuracy shape information.

For each media unit region of interest, the calculating of the higheraccuracy shape information may include virtually overlaying shapes ofcorresponding media units of interest of at least some of the matchingsignatures.

There may be provided a non-transitory computer readable medium forgenerating a hybrid representation of a media unit, the non-transitorycomputer readable medium may store instructions for receiving orgenerating the media unit; processing the media unit by performingmultiple iterations, wherein at least some of the multiple iterationsmay include applying, by spanning elements of the iteration, dimensionexpansion process that may be followed by a merge operation; selecting,based on an output of the multiple iterations, media unit regions ofinterest that contributed to the output of the multiple iterations; andproviding the hybrid representation, wherein the hybrid representationmay include shape information regarding shapes of the media unit regionsof interest, and a media unit signature that may include identifiersthat identify the media unit regions of interest.

The selecting of the media regions of interest may be executed persegment out of multiple segments of the media unit.

The shape information may include polygons that represent shapes thatsubstantially bound the media unit regions of interest.

The providing of the hybrid representation of the media unit may includecompressing the shape information of the media unit to providecompressed shape information of the media unit.

The non-transitory computer readable medium may store instructions forcomparing the media unit signature of the media unit to signatures ofmultiple concept structures to find a matching concept structure thathas at least one matching signature that matches to the media unitsignature; and calculating higher accuracy shape information that may berelated to regions of interest of the media unit, wherein the higheraccuracy shape information may be of higher accuracy than the compressedshape information of the media unit, wherein the calculating may bebased on shape information associated with at least some of the matchingsignatures.

The non-transitory computer readable medium may store instructions fordetermining shapes of the media unit regions of interest using thehigher accuracy shape information.

The for each media unit region of interest, the calculating of thehigher accuracy shape information may include virtually overlayingshapes of corresponding media units of interest of at least some of thematching signatures.

There may be provided a hybrid representation generator for generating ahybrid representation of a media unit, the hybrid representationgenerator may include an input that may be configured to receive orgenerate the media unit; a processor that may be configured to processthe media unit by by performing multiple iterations, wherein at leastsome of the multiple iterations may include applying, by spanningelements of the iteration, dimension expansion process that may befollowed by a merge operation; a selection unit that may be configuredto select, based on an output of the multiple iterations, media unitregions of interest that contributed to the output of the multipleiterations; and an output that may be configured to provide the hybridrepresentation, wherein the hybrid representation may include shapeinformation regarding shapes of the media unit regions of interest, anda media unit signature that may include identifiers that identify themedia unit regions of interest.

The selecting of the media regions of interest may be executed persegment out of multiple segments of the media unit.

The shape information may include polygons that represent shapes thatsubstantially bound the media unit regions of interest.

The hybrid representation generator that may be configured to compressthe shape information of the media unit to provide compressed shapeinformation of the media unit.

There may be provided a method for scale invariant object detection, themethod may include receiving or generating a first image in which anobject appears in a first scale and a second image in which the objectappears in a second scale that differs from the first scale; generatinga first image signature and a second image signature; wherein the firstimage signature may include a first group of at least one certain firstimage identifier that identifies at least a part of the object; whereinthe second image signature may include a second group of certain secondimage identifiers that identify different parts of the object; whereinthe second group may be larger than first group; and linking between theat least one certain first image identifier and the certain second imageidentifiers.

The method may include linking between the first image signature, thesecond image signature and the object.

The linking may include adding the first signature and the secondsignature to a certain concept structure that may be associated with theobject.

The method may include receiving or generating an input image;generating a signature of the input image; comparing the signature ofthe input image to signatures of the certain concept structure; anddetermining that the input image may include the object when at leastone of the signatures of the certain concept structure matches thesignature of the input image.

The method may include receiving or generating an input image;generating a signature of the input image, the signature of the inputimage may include only some of the certain second image identifiers;wherein the input image of the second scale; changing a scale of theinput image to the first scale to a provide an amended input image;generating a signature of the amended input image; and verifying thatthe input image may include the object when the signature of the amendedinput image may include the at least one certain first image identifier.

The method may include receiving or generating an input image;generating a signature of the input image; searching in the signature ofthe input image for at least one of (a) the at least one certain firstimage identifier, and (b) the certain second image identifiers; anddetermining that the input image may include the object when thesignature of the input image may include the at least one of (a) the atleast one certain first image identifier, and (b) the certain secondimage identifiers.

The method may include receiving or generating an input image;generating a signature of the input image; searching in the signature ofthe input image for at least one of (a) one or more certain first imageidentifier of the at least one certain first image identifier, and (b)at least one certain second image identifier of the certain second imageidentifiers; and determining that a input image includes the object whenthe signature of the input image may include the at least one of (a) oneor more certain first image identifier of the at least one certain firstimage identifier, and (b) the at least one certain second imageidentifier.

There may be provided a non-transitory computer readable medium forscale invariant object detection, the non-transitory computer readablemedium may store instructions for receiving or generating a first imagein which an object appears in a first scale and a second image in whichthe object appears in a second scale that differs from the first scale;generating a first image signature and a second image signature; whereinthe first image signature may include a first group of at least onecertain first image identifier that identifies at least a part of theobject; wherein the second image signature may include a second group ofcertain second image identifiers that identify different parts of theobject; wherein the second group may be larger than first group; andlinking between the at least one certain first image identifier and thecertain second image identifiers.

The non-transitory computer readable medium may store instructions forlinking between the first image signature, the second image signatureand the object.

The linking may include adding the first signature and the secondsignature to a certain concept structure that may be associated with theobject.

The non-transitory computer readable medium may store instructions forreceiving or generating an input image; generating a signature of theinput image; comparing the signature of the input image to signatures ofthe certain concept structure; and determining that the input image mayinclude the object when at least one of the signatures of the certainconcept structure matches the signature of the input image.

The non-transitory computer readable medium may store instructions forreceiving or generating an input image; generating a signature of theinput image, the signature of the input image may include only some ofthe certain second image identifiers; wherein the input image of thesecond scale; changing a scale of the input image to the first scale toa provide an amended input image; generating a signature of the amendedinput image; and verifying that the input image may include the objectwhen the signature of the amended input image may include the at leastone certain first image identifier.

The non-transitory computer readable medium may store instructions forreceiving or generating an input image; generating a signature of theinput image; searching in the signature of the input image for at leastone of (a) the at least one certain first image identifier, and (b) thecertain second image identifiers; and determining that a input imageincludes the object when the signature of the input image may includethe at least one of (a) the at least one certain first image identifier,and (b) the certain second image identifiers.

The non-transitory computer readable medium may store instructions forreceiving or generating an input image; generating a signature of theinput image; searching in the signature of the input image for at leastone of (a) one or more certain first image identifier of the at leastone certain first image identifier, and (b) at least one certain secondimage identifier of the certain second image identifiers; anddetermining that a input image includes the object when the signature ofthe input image may include the at least one of (a) one or more certainfirst image identifier of the at least one certain first imageidentifier, and (b) the at least one certain second image identifier.

There may be provided an object detector for scale invariant objectdetection, that may include an input that may be configured to receive afirst image in which an object appears in a first scale and a secondimage in which the object appears in a second scale that differs fromthe first scale; a signature generator that may be configured togenerate a first image signature and a second image signature; whereinthe first image signature may include a first group of at least onecertain first image identifier that identifies at least a part of theobject; wherein the second image signature may include a second group ofcertain second image identifiers that identify different parts of theobject; wherein the second group may be larger than first group; and anobject detection determination unit that may be configured to linkbetween the at least one certain first image identifier and the certainsecond image identifiers.

The object detection determination unit may be configured to linkbetween the first image signature, the second image signature and theobject.

The object detection determination unit may be configured to add thefirst signature and the second signature to a certain concept structurethat may be associated with the object.

The input may be configured to receive an input image; wherein thesignal generator may be configured to generate a signature of the inputimage; wherein the object detection determination unit may be configuredto compare the signature of the input image to signatures of the certainconcept structure, and to determine that the input image may include theobject when at least one of the signatures of the certain conceptstructure matches the signature of the input image.

The input may be configured to receive an input image; wherein thesignature generator may be configured to generate a signature of theinput image, the signature of the input image may include only some ofthe certain second image identifiers; wherein the input image of thesecond scale; wherein the input may be configured to receive an amendedinput image that was generated by changing a scale of the input image tothe first scale; wherein the signature generator may be configured togenerate a signature of the amended input image; and wherein the objectdetection determination unit may be configured to verify that the inputimage may include the object when the signature of the amended inputimage may include the at least one certain first image identifier.

The input may be configured to receive an input image; wherein thesignature generator may be configured to generate a signature of theinput image; wherein the object detection determination unit may beconfigured to search in the signature of the input image for at leastone of (a) the at least one certain first image identifier, and (b) thecertain second image identifiers; and determine that a input imageincludes the object when the signature of the input image may includethe at least one of (a) the at least one certain first image identifier,and (b) the certain second image identifiers.

The input may be configured to receive an input image; wherein thesignature generator may be configured to generate a signature of theinput image; wherein the object detection determination unit may beconfigured to search in the signature of the input image for at leastone of (a) one or more certain first image identifier of the at leastone certain first image identifier, and (b) at least one certain secondimage identifier of the certain second image identifiers; and determinethat a input image includes the object when the signature of the inputimage may include the at least one of (a) one or more certain firstimage identifier of the at least one certain first image identifier, and(b) the at least one certain second image identifier.

There may be provided a method for movement based object detection, themethod may include receiving or generating a video stream that mayinclude a sequence of images; generating image signatures of the images;wherein each image may be associated with an image signature that mayinclude identifiers; wherein each identifier identifiers a region ofinterest within the image; generating movement information indicative ofmovements of the regions of interest within consecutive images of thesequence of images; searching, based on the movement information, for afirst group of regions of interest that follow a first movement; whereindifferent first regions of interest may be associated with differentparts of an object; and linking between first identifiers that identifythe first group of regions of interest.

The linking may include linking between first image signatures thatinclude the first linked identifiers.

The linking may include adding the first image signatures to a firstconcept structure, the first concept structure may be associated withthe first image.

The method may include receiving or generating an input image;generating a signature of the input image; comparing the signature ofthe input image to signatures of the first concept structure; anddetermining that the input image may include a first object when atleast one of the signatures of the first concept structure matches thesignature of the input image.

The method may include receiving or generating an input image;generating a signature of the input image; searching in the signature ofthe input image for at least one of the first identifiers; anddetermining that the input image may include the object when thesignature of the input image may include at least one of the firstidentifiers.

The method may include generating location information indicative of alocation of each region of interest within each image; wherein thegenerating movement information may be based on the locationinformation.

There may be provided a non-transitory computer readable medium formovement based object detection, the non-transitory computer readablemedium may include receiving or generating a video stream that mayinclude a sequence of images; generating image signatures of the images;wherein each image may be associated with an image signature that mayinclude identifiers; wherein each identifier identifiers a region ofinterest within the image; wherein different region of interests includedifferent objects; generating movement information indicative ofmovements of the regions of interest within consecutive images of thesequence of images; searching, based on the movement information, for afirst group of regions of interest that follow a first movement; andlinking between first identifiers that identify the first group ofregions of interest.

The linking may include linking between first image signatures thatinclude the first linked identifiers.

The linking may include adding the first image signatures to a firstconcept structure, the first concept structure may be associated withthe first image.

The non-transitory computer readable medium may store instructions forreceiving or generating an input image; generating a signature of theinput image; comparing the signature of the input image to signatures ofthe first concept structure; and determining that the input image mayinclude a first object when at least one of the signatures of the firstconcept structure matches the signature of the input image.

The non-transitory computer readable medium may store instructions forreceiving or generating an input image; generating a signature of theinput image; searching in the signature of the input image for at leastone of the first identifiers; and determining that the input image mayinclude the object when the signature of the input image may include atleast one of the first identifiers.

The non-transitory computer readable medium may store instructions forgenerating location information indicative of a location of each regionof interest within each image; wherein the generating movementinformation may be based on the location information.

There may be provided an object detector that may include an input thatmay be configured to receive a video stream that may include a sequenceof images; a signature generator that may be configured to generateimage signatures of the images; wherein each image may be associatedwith an image signature that may include identifiers; wherein eachidentifier identifiers a region of interest within the image; a movementinformation unit that may be is configured to generate movementinformation indicative of movements of the regions of interest withinconsecutive images of the sequence of images; an object detectiondetermination unit that may be configured to search, based on themovement information, for a first group of regions of interest thatfollow a first movement; wherein different first regions of interest maybe associated with different parts of an object; and link between firstidentifiers that identify the first group of regions of interest.

The linking may include linking between first image signatures thatinclude the first linked identifiers.

The linking may include adding the first image signatures to a firstconcept structure, the first concept structure may be associated withthe first image.

The input may be configured to receive an input image; wherein thesignature generator may be configured to generate a signature of theinput image; and wherein the object detection determination unit may beconfigured to compare the signature of the input image to signatures ofthe first concept structure; and to determine that the input image mayinclude a first object when at least one of the signatures of the firstconcept structure matches the signature of the input image.

The input may be configured to receive an input image; wherein thesignature generator may be configured to generate a signature of theinput image; and wherein the object detection determination unit may beconfigured to search in the signature of the input image for at leastone of the first identifiers; and to determine that the input image mayinclude the object when the signature of the input image may include atleast one of the first identifiers.

The object detector that may be configured to generate locationinformation indicative of a location of each region of interest withineach image; wherein the generating of the movement information may bebased on the location information.

There may be provided a method for object detection, the method mayinclude receiving an input image; generating a signature of the inputimage; comparing the signature of the input image to signatures of aconcept structure; determining whether the signature of the input imagematches any of the signatures of the concept structure based onsignature matching criteria, wherein each signature of the conceptstructure may be associated within a signature matching criterion thatmay be determined based on an object detection parameter of thesignature; and concluding that the input image may include an objectassociated with the concept structure based on an outcome of thedetermining.

Each signature matching criterion may be determined by evaluating objectdetection capabilities of the signature under different signaturematching criteria.

The evaluating of the object detection capabilities of the signatureunder different signature matching criteria may include receiving orgenerating signatures of a group of test images; calculating the objectdetection capability of the signature, for each signature matchingcriterion of the different signature matching criteria; and selectingthe signature matching criterion based on the object detectioncapabilities of the signature under the different signature matchingcriteria.

The object detection capability reflects a percent of signatures of thegroup of test images that match the signature.

The selecting of the signature matching criterion may include selectingthe signature matching criterion that one applied results in a percentof signatures of the group of test images that match the signature thatmay be closets to a predefined desired percent of signatures of thegroup of test images that match the signature.

The signature matching criteria may be a minimal number of matchingidentifiers that indicate of a match.

There may be provided a method for managing a concept structure, themethod may include determining to add a new signature to the conceptstructure, wherein the concept structure already may include at leastone old signature; wherein the new signature may include identifiersthat identify at least parts of objects; and determining a new signaturematching criterion that may be based on one or more of the identifiersof the new signature; wherein the new signature matching criteriondetermines when another signature matches the new signature; wherein thedetermining of the new signature matching criterion may includeevaluating object detection capabilities of the signature underdifferent signature matching criteria.

The evaluating of the object detection capabilities of the signatureunder different signature matching criteria may include receiving orgenerating signatures of a group of test images; calculating the objectdetection capability of the signature, for each signature matchingcriterion of the different signature matching criteria; and selectingthe signature matching criterion based on the object detectioncapabilities of the signature under the different signature matchingcriteria.

The object detection capability reflects a percent of signatures of thegroup of test images that match the signature.

The selecting of the signature matching criterion may include selectingthe signature matching criterion that one applied results in a percentof signatures of the group of test images that match the signature thatmay be closets to a predefined desired percent of signatures of thegroup of test images that match the signature.

The signature matching criteria may be a minimal number of matchingidentifiers that indicate of a match.

There may be provided a non-transitory computer readable medium forobject detection, the non-transitory computer readable medium may storeinstructions for receiving an input image; generating a signature of theinput image; comparing the signature of the input image to signatures ofa concept structure; determining whether the signature of the inputimage matches any of the signatures of the concept structure based onsignature matching criteria, wherein each signature of the conceptstructure may be associated within a signature matching criterion thatmay be determined based on an object detection parameter of thesignature; and concluding that the input image may include an objectassociated with the concept structure based on an outcome of thedetermining.

Each signature matching criterion may be determined by evaluating objectdetection capabilities of the signature under different signaturematching criteria.

The evaluating of the object detection capabilities of the signatureunder different signature matching criteria may include receiving orgenerating signatures of a group of test images; calculating the objectdetection capability of the signature, for each signature matchingcriterion of the different signature matching criteria; and selectingthe signature matching criterion based on the object detectioncapabilities of the signature under the different signature matchingcriteria.

The object detection capability reflects a percent of signatures of thegroup of test images that match the signature.

The selecting of the signature matching criterion may include selectingthe signature matching criterion that one applied results in a percentof signatures of the group of test images that match the signature thatmay be closets to a predefined desired percent of signatures of thegroup of test images that match the signature.

The signature matching criteria may be a minimal number of matchingidentifiers that indicate of a match.

There may be provided a non-transitory computer readable medium formanaging a concept structure, the non-transitory computer readablemedium may store instructions for determining to add a new signature tothe concept structure, wherein the concept structure already may includeat least one old signature; wherein the new signature may includeidentifiers that identify at least parts of objects; and determining anew signature matching criterion that may be based on one or more of theidentifiers of the new signature; wherein the new signature matchingcriterion determines when another signature matches the new signature;wherein the determining of the new signature matching criterion mayinclude evaluating object detection capabilities of the signature underdifferent signature matching criteria.

The evaluating of the object detection capabilities of the signatureunder different signature matching criteria may include receiving orgenerating signatures of a group of test images; calculating the objectdetection capability of the signature, for each signature matchingcriterion of the different signature matching criteria; and selectingthe signature matching criterion based on the object detectioncapabilities of the signature under the different signature matchingcriteria.

The object detection capability reflects a percent of signatures of thegroup of test images that match the signature.

The selecting of the signature matching criterion may include selectingthe signature matching criterion that one applied results in a percentof signatures of the group of test images that match the signature thatmay be closets to a predefined desired percent of signatures of thegroup of test images that match the signature.

The signature matching criteria may be a minimal number of matchingidentifiers that indicate of a match.

There may be provided an object detector that may include an input thatmay be configured to receive an input image; a signature generator thatmay be configured to generate a signature of the input image; an objectdetection determination unit that may be configured to compare thesignature of the input image to signatures of a concept structure;determine whether the signature of the input image matches any of thesignatures of the concept structure based on signature matchingcriteria, wherein each signature of the concept structure may beassociated within a signature matching criterion that may be determinedbased on an object detection parameter of the signature; and concludethat the input image may include an object associated with the conceptstructure based on an outcome of the determining.

The object detector according to claim, may include a signature matchingcriterion unit that may be configured to determine each signaturematching criterion by evaluating object detection capabilities of thesignature under different signature matching criteria.

The input may be configured to receive signatures of a group of testimages; wherein the signature matching criterion unit may be configuredto calculate the object detection capability of the signature, for eachsignature matching criterion of the different signature matchingcriteria; and select the signature matching criterion based on theobject detection capabilities of the signature under the differentsignature matching criteria.

The object detector according to claim wherein the object detectioncapability reflects a percent of signatures of the group of test imagesthat match the signature.

The object detector according to claim wherein the signature matchingcriterion unit may be configured to select the signature matchingcriterion that one applied results in a percent of signatures of thegroup of test images that match the signature that may be closets to apredefined desired percent of signatures of the group of test imagesthat match the signature.

The object detector according to claim wherein the signature matchingcriteria may be a minimal number of matching identifiers that indicateof a match.

There may be provided a concept structure manager that may include acontroller that may be configured to determine to add a new signature tothe concept structure, wherein the concept structure already may includeat least one old signature; wherein the new signature may includeidentifiers that identify at least parts of objects; and a signaturematching criterion unit that may be configured to determine a newsignature matching criterion that may be based on one or more of theidentifiers of the new signature; wherein the new signature matchingcriterion determines when another signature matches the new signature;wherein the determining of the new signature matching criterion mayinclude evaluating object detection capabilities of the signature underdifferent signature matching criteria.

The signature matching criterion unit may be configured to determineeach signature matching criterion by evaluating object detectioncapabilities of the signature under different signature matchingcriteria.

The input may be configured to receive signatures of a group of testimages; wherein the signature matching criterion unit may be configuredto calculate the object detection capability of the signature, for eachsignature matching criterion of the different signature matchingcriteria; and select the signature matching criterion based on theobject detection capabilities of the signature under the differentsignature matching criteria.

The concept manager according to claim wherein the object detectioncapability reflects a percent of signatures of the group of test imagesthat match the signature.

The concept manager according to claim wherein the signature matchingcriterion unit may be configured to select the signature matchingcriterion that one applied results in a percent of signatures of thegroup of test images that match the signature that may be closets to apredefined desired percent of signatures of the group of test imagesthat match the signature.

Obstacle Avoidance

It will be appreciated that in “normal”, non-assisted driving scenarios,a driver may encounter driving obstacles that are not necessarilyindicated in commonly available roadmaps. For example, it is notuncommon for a roadway to have potholes that may complicate the drivingprocess. Some potholes are large enough to cause damage to the wheels orundercarriage of the vehicle. Accordingly, when encountering a pothole,the driver may swerve to avoid it. However, depending on the layout ofthe roadway and/or traffic conditions, it may not be feasible for thedriver to react in such a manner. It is also possible that depending oncurrent visibility, the speed of the vehicle, and/or the alertness ofthe driver, the driver may not even see the pothole in time to avoid it.

It will be appreciated that such issues are also relevant in anassisted/autonomous driving scenario. As discussed hereinabove, suchsystems may use one or more sensors to acquire information about thecurrent driving environment in order to determine how to drive along theroadway as per a stored map. It is possible, that the vehicle'ssensor(s) may not detect an obstacle, e.g., a pothole, sufficiently inadvance in order to enable the system to determine that evasive actionshould be taken to avoid the obstacle and to perform that action intime. It will be appreciated that such a scenario may be furthercomplicated by the presence of other vehicles and/or pedestrians in theimmediate vicinity of the vehicle.

Reference is now made to FIG. 3A, which is a partly-pictorial,partly-block diagram illustration of an exemplary obstacle detection andmapping system 10 (hereinafter referred to also as “system 10”)constructed and operative in accordance with embodiments describedherein. As described herein, system 10 may be operative to processsensor data to detect and map obstacles on a roadway in order to enablean assisted/autonomous driving system to predict and avoid obstacles inthe path of a vehicle.

System 10 comprises vehicle 100 and obstacle avoidance server 400 whichmay be configured to communicate with each other over a communicationsnetwork such as, for example, the Internet. In accordance with theexemplary embodiment of FIG. 3A, vehicle 100 may be configured with anautonomous driving system (not shown) operative to autonomously providedriving instructions to vehicle 100 without the intervention of a humandriver. It will be appreciated that the embodiments described herein mayalso support the configuration of vehicle 100 with an assisted (or“semi-autonomous”) driving system where in at least some situations ahuman driver may take control of vehicle 100 and/or where in at leastsome situations the semi-autonomous driving system provides warnings tothe driver without necessarily directly controlling vehicle 100.

In accordance with the exemplary embodiment of FIG. 3A, vehicle 100 maybe configured with at least one sensor 130 to provide information abouta current driving environment as vehicle 100 proceeds along roadway 20.It will be appreciated that while sensor 130 is depicted in FIG. 3A as asingle entity, in practice, as will be described hereinbelow, there maybe multiple sensors 130 arrayed on, or inside of, vehicle 130. Inaccordance with embodiments described herein, sensor(s) 130 may beimplemented using a conventional camera operative to capture images ofroadway 20 and objects in its immediate vicinity. It will be appreciatedthat sensor 130 may be implemented using any suitable imaging technologyinstead of, or in addition to, a conventional camera. For example,sensor 130 may also be operative to use infrared, radar imagery,ultrasound, electro-optics, radiography, LIDAR (light detection andranging), etc. Furthermore, in accordance with some embodiments, one ormore sensors 130 may also be installed independently along roadway 20,where information from such sensors 130 may be provided to vehicle 100and/or obstacle avoidance server 400 as a service.

In accordance with the exemplary embodiment of FIG. 3A, static referencepoints 30A and 30B (collectively referred to hereinafter as staticreference points 30) may be located along roadway 20. For example,static reference point 30A is depicted as a speed limit sign, and staticreference point 30B is depicted as an exit sign. In operation, sensor130 may capture images of static reference points 30. The images maythen be processed by the autonomous driving system in vehicle 100 toprovide information about the current driving environment for vehicle100, e.g., the speed limit or the location of an upcoming exit.

Obstacle 40, e.g., a pothole, may be located on roadway 20. Inaccordance with embodiments described herein, sensor 130 may also beoperative to capture images of obstacle 40. The autonomous drivingsystem may be operative to detect the presence of obstacle 40 in theimages provided by sensor 130 and to determine an appropriate response,e.g., whether or not vehicle 100 should change speed and/or direction toavoid or minimize the impact with obstacle 40. For example, if theautonomous driving system determines that obstacle 40 is a piece ofpaper on roadway 20, no further action may be necessary. However, ifobstacle 40 is a pothole, the autonomous driving system may instructvehicle 100 to slow down and/or swerve to avoid obstacle 40.

It will be appreciated that there may not be sufficient processing timefor the autonomous driving system to determine an appropriate response;depending on the speed of vehicle 100 and the distance at which sensor130 captures an image of obstacle 40, vehicle 100 may drive over/throughobstacle 40 (or collide with it) before obstacle 40 is detected and/oran appropriate response may be determined by the autonomous drivingsystem. In accordance with embodiments described herein, in order toprovide additional processing time for obstacle detection and responsedetermination, vehicle 100 may exchange obstacle information withobstacle avoidance server 400. The autonomous driving system may beoperative to upload the imagery received from sensor 130 and/or thedetermination of the nature of obstacle 40 to obstacle avoidance server400, and obstacle avoidance server 400 may be operative to provideobstacle information received in such manner from other vehicles 100 inorder to enable the autonomous driving system to anticipate and respondto obstacles 40 in advance of, or in parallel to, receiving the relevantimagery from sensor 130.

Depending on the configuration of system 10 and vehicle 100, obstacleavoidance server 400 and vehicle 100 may exchange obstacle informationin real-time (or near real-time) and/or in a burst mode, where theobstacle information may be provided either periodically and/or whenconditions are suitable for a data exchange between obstacle avoidanceserver 400 and vehicle 100. For example, vehicle 100 may be configuredto perform regular uploads/downloads of obstacle information when theengine is turned on (or off, with a keep alive battery function tofacilitate communication with obstacle avoidance server 400).Alternatively, or in addition, vehicle 100 may be configured to performsuch uploads/downloads when stationary (e.g., parked, or waiting at atraffic light) and a sufficiently reliable wireless connection isdetected. Alternatively, or in addition, the uploads/downloads may betriggered by location, where obstacle information may be downloaded fromobstacle avoidance server 400 when vehicle 100 enters an area for whichit does not have up-to-date obstacle information.

It will also be appreciated that in some situations, the autonomousdriving system may not detect obstacle 40 at all; it is possible thatvehicle 100 may pass by/over/through obstacle 40 without detecting it.The autonomous driving system may have finite resources available toanalyze imagery from sensor 130 in a finite period of time in order todetermine a currently relevant driving policy. It is understandable thatif, for whatever reason (e.g., driving speed, visibility, etc.), theautonomous driving system does not detect obstacle 40 in real-time ornear-real time there may be little benefit to be realized fromcontinuing processing the imagery on vehicle 100. However, the detectionof a “missed” obstacle 40 may be of benefit to another vehicle 100 onroadway 20.

Accordingly, in some configurations of system 10, vehicle 100 may alsobe configured to upload raw driving data from sensor 130 for processingby obstacle avoidance server 400 to detect obstacles 40. Obstacleinformation associated with obstacles 40 detected in such manner maythen be provided by obstacle avoidance server 400 as describedhereinabove.

It will also be appreciated that it may be possible to use non-imagedata to detect obstacles 40 from the raw driving data. For example, atleast one sensor 130 may be implemented as an electronic control unit(ECU) controlling the shock absorbers of vehicle 100. Obstacle avoidanceserver 400 may be configured to determine that one or more shocks of acertain magnitude and/or pattern registered by the shock absorber ECUmay be indicative of a pothole.

In accordance with some embodiments described herein, the sources forraw driving data to be used by obstacle avoidance server 400 to detectobstacles 40 may not be limited to just vehicles 100 with an autonomousdriving system. Obstacle avoidance server 400 may also receive rawdriving data from vehicles that are being driven manually and/or withthe assistance of a semi-autonomous driving system. It will beappreciated that in for such vehicles, the raw driving data may includeindications of driver recognition of obstacles 40. For example, when adriver detects obstacle 40, e.g., sees a pothole, the driver may slowdown in approach of obstacle 40, swerve to avoid it, and/or even changelanes. Accordingly, the raw driving data processed by obstacle avoidanceserver 400 may also include raw data uploaded from vehicles without anautonomous driving system.

Obstacle avoidance server 400 may use supervised and/or unsupervisedlearning methods to detect obstacles 40 in the raw driving data;reference data sets for the detection of obstacles 40 and thedetermination of associated driving policies may be prepared based onunsupervised analysis of raw driving data and/or using manual labellingand extraction.

It will be appreciated that it may not be a trivial task to pinpoint alocation for obstacle 40. While it may generally be assumed that vehicle100 is configured with a location-based service, e.g., globalpositioning satellite (GPS) system, such systems may not necessarily beaccurate enough to provide a sufficient resolution regarding theexpected location for obstacle 40. For example, one possible drivingpolicy for evasive action may entail avoiding obstacle 40 by swerving orchanging lanes on roadway 20. If the location-based service is onlyaccurate to within five feet, swerving within the lane may actuallyincrease the likelihood of vehicle 100 hitting the pothole. Similarly,it may not be possible to determine in which lane obstacle 40 islocated.

In accordance with embodiments described herein, static reference points30 may be used in system 10 to more accurately determine a location forobstacle 40; the location of obstacle 40 may be defined as an offsetfrom a fixed location for one of static reference points 30. Forexample, in the exemplary embodiment of FIG. 3A, the location ofobstacle 40 may be defined as “x” feet past static reference point 30A,where the distance in feet may be calculated as a function of the framesper second of the camera used for sensor 130 and the speed of vehicle100. Alternatively, or in addition, two sensors 130 may be employed touse triangulation to determine the distance between static referencepoint 30A and obstacle 40. Alternatively, or in addition, the distancemay be calculated using both reference points 30A and 30B totriangulate.

It will be appreciated that as described hereinabove, obstacle 40 may bea “persistent obstacle” in that it may be reasonably expected to remainon roadway 20 for an extended period of time. For example, potholestypically remain (and even increase in size) for several months beforebeing repaired. However, some obstacles may be more transient in nature.For example, a tree fallen on the road or debris from a traffic accidentmay typically be removed from roadway 20 within a few hours. Suchtransient obstacles may, at least in some cases, be easier to detectthan persistent obstacles, e.g., a fallen tree is easier to identifyingfrom an approaching vehicle than a pothole.

It will therefore be appreciated that the driving policies for transientobstacles may be different than for persistent obstacles. For example,in a case where obstacle 40 is a pothole that may be assumed to bepersistent, the associated driving policy may entail the autonomousdriving system “erring on the side of caution” by performing evasiveaction (e.g., swerving or changing lanes) even if the pothole isn'tdetected in the data provided by sensor 130 when vehicle 100 reaches thepothole's expected location. However, in a case where obstacle 40 is atree on roadway 20, the associated driving policy may entail theautonomous driving system performing preventative action, e.g., slowingdown, prior to reaching the tree's expected location, but not to performevasive action if the tree isn't actually detected according to the dataprovided by sensor 130.

It will also be appreciated that some obstacles may be “recurringobstacles” where transient obstacles recur in the same location. Forexample, recurring obstacle 50 represents a slight depression in roadway20. The impact on vehicle 100 as it drives over a slight depression inroadway 20 may be minimal. However, in some cases it may be observedthat the slight depression may fill with water or mud when it rains, orintermittently ice over during the winter. Accordingly, the drivingpolicies for recurring obstacles 50 may more heavily weight otherenvironmental factors, e.g., season or weather, than the drivingpolicies for obstacles 40.

Reference is now made to FIG. 3B which is a block diagram of anexemplary autonomous driving system 200 (hereinafter also referred to assystem 200), constructed and implemented in accordance with embodimentsdescribed herein. Autonomous driving system 200 comprises processingcircuitry 210, input/output (I/O) module 220, camera 230, telemetry ECU240, shock sensor 250, autonomous driving manager 260, and obstacledatabase 270. Autonomous driving manager 260 may be instantiated in asuitable memory for storing software such as, for example, an opticalstorage medium, a magnetic storage medium, an electronic storage medium,and/or a combination thereof. It will be appreciated that autonomousdriving system 200 may be implemented as an integrated component of anonboard computer system in a vehicle, such as, for example, vehicle 100from FIG. 3A. Alternatively, system 200 may be implemented and aseparate component in communication with the onboard computer system. Itwill also be appreciated that in the interests of clarity, whileautonomous driving system 200 may comprise additional components and/orfunctionality e.g., for autonomous driving of vehicle 100, suchadditional components and/or functionality are not depicted in FIG. 3Band/or described herein.

Processing circuitry 210 may be operative to execute instructions storedin memory (not shown). For example, processing circuitry 210 may beoperative to execute autonomous driving manager 260. It will beappreciated that processing circuitry 210 may be implemented as acentral processing unit (CPU), and/or one or more other integratedcircuits such as application-specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), full-custom integrated circuits, etc.,or a combination of such integrated circuits. It will similarly beappreciated that autonomous driving system 200 may comprise more thanone instance of processing circuitry 210. For example, one such instanceof processing circuitry 210 may be a special purpose processor operativeto execute autonomous driving manager 260 to perform some, or all, ofthe functionality of autonomous driving system 200 as described herein.

I/O module 220 may be any suitable communications component such as anetwork interface card, universal serial bus (USB) port, disk reader,modem or transceiver that may be operative to use protocols such as areknown in the art to communicate either directly, or indirectly, withother elements of system 10 (FIG. 3A) and/or system 200, such as, forexample, obstacle avoidance server 400 (FIG. 3A), camera 230, telemetryECU 240, and/or shock sensor 250. As such, I/O module 220 may beoperative to use a wired or wireless connection to connect to obstacleavoidance server 400 via a communications network such as a local areanetwork, a backbone network and/or the Internet, etc. I/O module 220 mayalso be operative to use a wired or wireless connection to connect toother components of system 200, e.g., camera 230, telemetry ECU 240,and/or shock sensor 250. It will be appreciated that in operation I/Omodule 220 may be implemented as a multiplicity of modules, wheredifferent modules may be operative to use different communicationtechnologies. For example, a module providing mobile networkconnectivity may be used to connect to obstacle avoidance server 400,whereas a local area wired connection may be used to connect to camera230, telemetry ECU 240, and/or shock sensor 250.

In accordance with embodiments described herein, camera 230, telemetryECU 240, and shock sensor 250 represent implementations of sensor(s) 130from FIG. 3A. It will be appreciated that camera 230, telemetry ECU 240,and/or shock sensor 250 may be implemented as integrated components ofvehicle 100 (FIG. 3A) and may provide other functionality that is theinterests of clarity is not explicitly described herein. As describedhereinbelow, system 200 may use information about a current drivingenvironment as received from camera 230, telemetry ECU 240, and/or shocksensor 250 to determine an appropriate driving policy for vehicle 100.

Autonomous driving manager 260 may be an application implemented inhardware, firmware, or software that may be executed by processingcircuitry 210 to provide driving instructions to vehicle 100. Forexample, autonomous driving manager 260 may use images received fromcamera 230 and/or telemetry data received from telemetry ECU 240 todetermine an appropriate driving policy for arriving at a givendestination and provide driving instructions to vehicle 100 accordingly.It will be appreciated that autonomous driving manager 260 may also beoperative to use other data sources when determining a driving policy,e.g., maps of potential routes, traffic congestion reports, etc.

As depicted in FIG. 3B, autonomous driving manager 260 comprisesobstacle detector 265, obstacle predictor 262, and obstacle avoidancemodule 268. It will be appreciated that the depiction of obstacledetector 265, obstacle predictor 262, and obstacle avoidance module 268as integrated components of autonomous driving manager 260 may beexemplary. The embodiments described herein may also supportimplementation of obstacle detector 265, obstacle predictor 262, andobstacle avoidance module 268 as independent applications incommunication with autonomous driving manager 260, e.g., via I/O module220.

Obstacle detector 265, obstacle predictor 262, and obstacle avoidancemodule 268 may be implemented in hardware, firmware, or software and maybe invoked by autonomous driving manager 260 as necessary to provideinput to the determination of an appropriate driving policy for vehicle100. For example, obstacle detector 265 may be operative to useinformation from sensor(s) 130 (FIG. 3A), e.g., camera 230, telemetryECU 240, and/or shock sensor 250 to detect obstacles in (or near) thedriving path of vehicle 100, e.g., along (or near) roadway 20 (FIG. 3A).Obstacle predictor 262 may be operative to use obstacle informationreceived from obstacle avoidance server 400 to predict the location ofobstacles along or near roadway 20 before, or in parallel to theirdetection by obstacle detector 265. Obstacle avoidance module 268 may beoperative to determine an appropriate driving policy based at least onobstacles detected/predicted (or not detected/predicted) by obstacledetector 265 and/or obstacle predictor 262.

Autonomous driving manager 260 may store obstacle information receivedfrom obstacle avoidance server 400 in obstacle database 270 for use byobstacle detector 265, obstacle predictor 262, and obstacle avoidancemodule 268 as described herein. It will be appreciated that obstacleinformation, including driving policies to avoid detected obstacles mayalso be stored in obstacle database 270 for use by obstacle detector265, obstacle predictor 262, and obstacle avoidance module 268.

Reference is now made also to FIG. 3C which is a flowchart of anexemplary obstacle detection and avoidance process 300 (hereinafter alsoreferred to as process 300) to be performed by system 200 during (orpreceding) a driving session for vehicle 100 (FIG. 3A). The componentsof systems 100 and 200 will be referred to herein as per the referencenumerals in FIGS. 3A and 3B.

In accordance with some embodiments described herein, system 200 mayreceive (step 310) an obstacle warning from obstacle avoidance server400, e.g. obstacle information received from other vehicles 100 usingsystem 10 and/or and other suitable sources as described with referenceto FIG. 3A. It will be appreciated that the obstacle warning may bereceived by I/O module 220. Depending on the configuration of system100, the obstacle warning may be received in a batch update process,either periodically and/or triggered by an event, e.g., when vehicle 100is turned on, when vehicle 100 enters a new map area, when vehicle 100enters an area with good wireless reception, etc. It will be appreciatedthat the obstacle warning received in step 310 may also include adescription of the obstacle, e.g., a pothole.

Autonomous driving manager 260 may invoke obstacle predictor 262 toestimate (step 320) a location for the obstacle associated with theobstacle warning. It will be appreciated that, as discussed hereinabove,while the obstacle warning may include GPS coordinates for the obstacle,the coordinates may not be accurate enough to provide the location withsufficient precision to be of use in an automated. Furthermore, even ifthe coordinates for the obstacle, e.g., obstacle 40 (FIG. 3A) areaccurate, vehicle 10 will likely be in movement and it may not bepossible to accurately determine the GPS coordinates of vehicle 10 as itapproaches the obstacle.

In accordance with embodiments described herein, the obstacleinformation in the obstacle warning may include landmarks to be used forcalculating the obstacle location. For example, the location of theassociated obstacle may be indicated in the obstacle warning as anoffset from a local landmark, e.g., one or more of static referencepoints 30 (FIG. 3A). It will be appreciated that static reference points30 may be visible from a greater distance than obstacle 40; a signposttypically extends vertically from the ground such that static referencepoints 30 may be captured by sensor(s) 130, e.g., camera 230, in advanceof detection of obstacle 40. Obstacle predictor may then use staticreference points 30 to predict a location for the associated obstacle 40as an offset from the observed location of one or more static referencepoints 30 before sensor(s) 130 may actually provide an indication of thelocation. Distance from a given static reference point 30 may beestimated as a function of current speed for vehicle 10 and aframes-per-second setting for camera 230. Alternatively, or in addition,multiple static reference points may be used to locate obstacle 40 bytriangulation, e.g., a distance from both static reference points 30Aand 30B may be estimated and then used to predict the location ofobstacle 40.

It will be appreciated that in some embodiments, sensor(s) 130 may beoperative to provide an estimate for distance. For example, sensor(s)may be operative to use radar imagery, LIDAR, etc.

Autonomous driving manager 260 may invoke obstacle avoidance module 268to enact (step 330) preventative measures to avoid colliding with, orrunning over, obstacle 40. Obstacle avoidance module 268 may enact thepreventative measures in accordance with driving policies stored inobstacle database 270. It will be appreciated that the obstacle warningmay include a description of obstacle 40, e.g., a pothole. Using theobstacle description and other available information (e.g., distance toobstacle 40 as estimated in step 320, current speed, road conditions,etc.), obstacle avoidance module 268 may select an appropriate drivingpolicy from obstacle database 270 for implementation. For example, theselected driving policy may entail a reduction in speed, switching lanes(if possible under current conditions), swerving within a lane, etc. Itwill be appreciated that the driving policies in obstacle database maybe may be pre-stored in obstacle database 270. It will further beappreciated that the driving policies may be derived based on supervisedor unsupervised analysis of imagery from actual driving sessions. Some,or all, of the driving policies may also be defined manually.

In some cases, the selected driving policy may entail more extremepreventative measures. For example, obstacle 40 may be described in theobstacle warning as a very large obstacle that may effectively blocktraffic on roadway 20, e.g., a rockfall, or debris from a multi-vehicleaccident. In such a case, the driving policy may entail actions such as,for example, pulling over on the shoulder, stopping in place, orchanging routes.

Autonomous driving manager may invoke obstacle detector 265 to determineif an obstacle, e.g., obstacle 40, has been detected (step 340). Forexample, obstacle detector 265 may use imagery received from sensor(s)130 to detect obstacle 40 in roadway 20.

It will be appreciated that in typical operation of process 300, steps310-330 may not always be performed; autonomous driving system 200 maynot always receive an obstacle warning in advance of encounteringobstacle 40. For example, vehicle 100 may be the first vehicleassociated with system 10 to encounter obstacle 40. It will therefore beappreciated that autonomous driving manager 260 may continuously executeprocess 300 in an attempt to detect and avoid obstacles as vehicle 100progresses along roadway 20. Accordingly, step 340 may be performedindependently of whether or not an obstacle warning has been receivedfor a given area on roadway 20. It will, however, be appreciated, thatin the event that an obstacle warning was indeed received, the relevantobstacle information may be used by obstacle detector 265 to determinewhether or not the indicated obstacle, e.g., obstacle 40, is indeed inplace on roadway 20 as indicated in the obstacle warning.

Alternatively, or in addition, obstacle detector 265 may be detected inresponse to information received from shock sensor 250. Shock sensor 250may be a sensor operative to monitor the operation of the shockabsorbers on vehicle 100. It will be appreciated that when running overa pothole, shock absorbers may absorb the ensuing shock and stabilizevehicle 100. It will also be appreciated that in some situations,potholes may not be detected until vehicle 100 actually drives overthem. In such a situation, obstacle detector 265 may use informationfrom shock sensor 250 to detect obstacle 40, albeit after vehicle 100has already hit obstacle 40.

If obstacle 40 is detected in step 340, autonomous driving manager 260may invoke obstacle avoidance module 268 to determine whether obstacle40 is in the path of vehicle 100 (step 350). It will be appreciated thatsome obstacles 40 on roadway 20 may not be in the direct path of vehicle100; some obstacles 40 may be in a different lane than vehicle 100.Additionally, if step 330 was performed, vehicle 100 may have alreadyswitched lanes in advance of obstacle 40 being detected in step 340.Furthermore, if obstacle 40 was detected using shock sensor 250, vehicle100 may have already run over obstacle 40 such that the result of step350 may be “no”. If obstacle 40 is not detected in step 340, processcontrol may flow through to step 380.

If, as per the result of step 350, obstacle 40 is in the path of vehicle100, obstacle avoidance module 268 may determine whether obstacle 40 isavoidable (step 360). It will be appreciated that obstacle 40 may not bedetected in step 340 until vehicle 100 is very close. For example,visibility may be poor, obstacle 40 may be located immediately around abend in roadway 20, obstacle 40 may be effectively camouflaged by itssurroundings (e.g., some potholes are not distinguishable from adistance), etc. Accordingly, by the time obstacle 40 is detected, theremay not be enough time to enact avoidance measures prior to impact. If,as per the result of step 350, obstacle 40 is not avoidable, processcontrol may flow through to step 380.

If, as per the result of step 360, obstacle 40 is avoidable, obstacleavoidance module 268 may instruct vehicle 100 to avoid (step 370)obstacle 40. For example, obstacle avoidance module 268 may instructvehicle 100 to straddle obstacle 40, swerve within the lane, or switchlanes altogether.

If, as per the result of step 360, obstacle 40 is not avoidable,obstacle avoidance module 268 may instruct vehicle 100 to minimize (step365) impact with obstacle 40. For example, obstacle avoidance module 268may instruct vehicle 100 to reduce speed and/or to swerve to avoidhitting the center of obstacle 40. Process control may then flow throughto step 380.

Autonomous driving manager 380 may use I/O module 220 to send anobstacle report to obstacle avoidance server 400. The obstacle reportmay, for example, include telemetry data and sensor imagery that may beused by obstacle avoidance server 400 to determine a location forobstacle 40. For example, the obstacle report may include camera images(from camera 230) and telemetry data (from telemetry ECU 240) from aperiod staring a given number of seconds before obstacle 40 was detectedin step 340. Alternatively, or in addition, the telemetry data andcamera images may be from a period defined according to a physicaldistance from obstacle 40. The obstacle report may also include datafrom shock sensor 250 where applicable.

Depending on the configuration of systems 100 and 200, step 380 may beperformed in real time or near real time after obstacle 40 is detectedin step 340. Alternatively, or in addition, step 380 may be performed ina periodic and/or event triggered batch process.

It will be appreciated that obstacle reports may also be provided for“no” results in process 300. If obstacle 40 is not detected in step 340,an obstacle report may be provided to obstacle avoidance server 400indicating that an obstacle was not detected at the associated location.It will be appreciated that in some cases, a “no” result in step 340 mayindicate that an obstacle associated with an obstacle warning may nolonger be found on roadway 20, e.g., the pothole may have been filledin. In other cases, where an obstacle warning was not received, the “no”result may indicate that the default condition, e.g., “no obstacles”, isin effect for roadway 20.

Reference is now made to FIG. 4 which is a block diagram of an exemplaryobstacle avoidance server 400 (hereinafter also referred to as server400), constructed and implemented in accordance with embodimentsdescribed herein. Server 400 comprises processing circuitry 410,input/output (I/O) module 420, obstacle avoidance manager 460, andobstacle database 470. Obstacle avoidance manager 460 may beinstantiated in a suitable memory for storing software such as, forexample, an optical storage medium, a magnetic storage medium, anelectronic storage medium, and/or a combination thereof.

Processing circuitry 410 may be operative to execute instructions storedin memory (not shown). For example, processing circuitry 410 may beoperative to execute obstacle avoidance manager 260. It will beappreciated that processing circuitry 410 may be implemented as acentral processing unit (CPU), and/or one or more other integratedcircuits such as application-specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), full-custom integrated circuits, etc.,or a combination of such integrated circuits. It will similarly beappreciated that server 400 may comprise more than one instance ofprocessing circuitry 410. For example, one such instance of processingcircuitry 410 may be a special purpose processor operative to executeobstacle avoidance manager 460 to perform some, or all, of thefunctionality of server 400 as described herein.

I/O module 420 may be any suitable communications component such as anetwork interface card, universal serial bus (USB) port, disk reader,modem or transceiver that may be operative to use protocols such as areknown in the art to communicate either directly, or indirectly, withother elements of system 10 (FIG. 3A) such as, for example, system 200(FIG. 3B). As such, I/O module 420 may be operative to use a wired orwireless connection to connect to system 200 via a communicationsnetwork such as a local area network, a backbone network and/or theInternet, etc. It will be appreciated that in operation I/O module 220may be implemented as a multiplicity of modules, where different modulesmay be operative to use different communication technologies. Forexample, a module providing mobile network connectivity may be used toconnect wirelessly to one instance of system 200, e.g., one vehicle 100(FIG. 3A), whereas a local area wired connection may be used to connectto a different instance of system 100, e.g., a different vehicle 100.

Obstacle avoidance manager 460 may be an application implemented inhardware, firmware, or software that may be executed by processingcircuitry 410 to provide obstacle warnings and/or driving policies tovehicles 100. For example, obstacle avoidance manager 460 may useobstacle information in obstacle reports received from vehicles 100 toprovide obstacle warnings in real time, near real time, and/or batchmode to vehicles 100 in system 10 (FIG. 3A). It will be appreciated thatobstacle avoidance manager 460 may also be operative to use other datasources when detecting obstacles and determining driving policies, e.g.,maps of potential routes, traffic congestion reports, etc.

As depicted in FIG. 4, obstacle avoidance manager 460 comprises obstacleextractor 462, obstacle categorizer 464, obstacle policy manager 466,and obstacle timer 468. It will be appreciated that the depiction ofobstacle extractor 462, obstacle categorizer 464, obstacle policymanager 466, and obstacle timer 468 as integrated components of obstacleavoidance manager 460 may be exemplary. The embodiments described hereinmay also support implementation of obstacle extractor 462, obstaclecategorizer 464, obstacle policy manager 466, and obstacle timer 468 asindependent applications in communication with obstacle avoidancemanager 460, e.g., via I/O module 420.

Obstacle extractor 462, obstacle categorizer 464, obstacle policymanager 466, and obstacle timer 468 may be implemented in hardware,firmware, or software and may be invoked by obstacle avoidance manager460 as necessary to provide obstacle warnings and associated drivingpolicies to vehicles 100. For example, obstacle extractor 462 may beoperative to extract obstacle information from obstacle reports receivedfrom vehicles 100. Obstacle categorizer 464 may be operative to usecategorize obstacles in terms of danger level and/or persistencecharacteristics. Obstacle policy manager 466 may be operative todetermine an appropriate driving policy for a given obstacle or categoryof obstacle. Obstacle timer 468 may be operative to set an expirationtimer for obstacles according to their associated obstacle categories.

Obstacle avoidance manager 460 may store obstacle information receivedfrom vehicles 100 in obstacle database 270 for use by obstacle extractor462, obstacle categorizer 464, obstacle policy manager 466, and obstacletimer 468 as described herein. It will be appreciated that obstacledatabase 470 may also be used to store obstacle information, includingdriving policies, derived by obstacle extractor 462, obstaclecategorizer 464, obstacle policy manager 466, and obstacle timer 468.

Reference is now made also to FIG. 5 which is a flowchart of anexemplary obstacle detection and warning process 500 (hereinafter alsoreferred to as process 500) to be performed by server 400 to supportdriving sessions for vehicles 100 (FIG. 3A). The components of systems10 and 200 will be referred to herein as per the reference numerals inFIGS. 3A and 3B. 1 and 2. Similarly, the components of server 400 willbe referred to herein as per the reference numerals in FIG. 4.

In accordance with some embodiments described herein, server 400 mayreceive (step 510) an obstacle report from vehicle 100. It will beappreciated that the obstacle warning may be received by I/O module 420.Depending on the configuration of system 10, the obstacle report may bereceived in real time or near real time after vehicle 100 encountersobstacle 40. Alternatively, or in addition, the obstacle report may bereceived in a batch update process, either periodically and/or triggeredby an event, e.g., when vehicle 100 is turned on, when vehicle 100enters a new map area, when vehicle 100 enters an area with goodwireless reception, etc. It will be appreciated that the obstacle reportreceived in step 510 may also include an image of obstacle 40.

It will be appreciated that as described hereinabove with respect toprocess 300 (FIG. 3C), a given obstacle report may not necessarily betriggered by detection of an obstacle by vehicle 100. The obstaclereport may represent a periodic upload of GPS data, camera images,telemetry data, and/or other sensor data that may not be associated witha specific detection of an obstacle. The obstacle report may also beassociated with a non-detection of an obstacle that was otherwiseindicated by a previously issued obstacle warning. Furthermore, asdescribed with respect to FIG. 3A, raw driving data from manually and/orsemi-autonomously driven vehicles may also be provided to server 400. Inaccordance with embodiments described herein, such raw driving data mayalso be input to process 500 along with obstacle reports.

Accordingly, obstacle avoidance server 400 may invoke obstacle extractor462 to analyze received obstacle reports and/or raw driving data todetermine if a given obstacle report (or raw driving data) includes areported obstacle (step 520), i.e., where the obstacle reportspecifically addresses an obstacle, whether it was detected by vehicle100 and/or it was included in a previous obstacle warning.

Obstacle extractor 462 may determine if the obstacle report or rawdriving data includes at least a reference to a reported obstacle (step520). It will be appreciated that as described with respect to process300, an obstacle report may be sent in response to an obstacle warning,even if the associated obstacle is not actually detected by autonomousdriving manager 360. Such an obstacle report may still include areference to the reported obstacle. If the obstacle report or rawdriving data includes at least a reference to a reported obstacle (step520) processing may continue with step 530.

If the obstacle report or raw driving data does not include a referenceto a reported obstacle (step 520), obstacle extractor 462 may process(step 525) the obstacle report (or raw driving data) to extract obstacleinformation for obstacles. For example, obstacle extractor 464 may beoperative to detect an obstacle based on an image of an obstacle, wherethe image may be matched to a reference image of an obstacle stored inobstacle database 470. Alternatively, or in addition, obstacle extractor464 may be operative to detect an obstacle based on telemetry data,where certain driving patterns may be indicative of a driver avoiding anobstacle. Alternatively, or in addition, obstacle extractor 464 may beoperative to detect an obstacle based on analysis of shock absorberdata, where shock events may be indicative of a vehicle running over apothole. It will be appreciated that the obstacle information extractedin step 525 may also include reference data that may be used to providea location for the extracted obstacle, e.g., GPS coordinates, offsetsfrom fixed reference points 30, radar imagery, etc.

If one or more obstacles are extracted (step 529) processing control mayflow through to step 530. Otherwise, processing control may return tostep 510.

Obstacle avoidance manager 460 may determine whether a reported obstacleor an extracted obstacle is a “new” obstacle (step 530), i.e., whetherthe associated obstacle is already known to server 400 from a previousiteration of process 300. If the obstacle is a new obstacle, obstacleavoidance manager 460 may invoke obstacle categorizer 464 to categorizethe obstacle. Obstacle categorizer 464 may categorize the obstacle basedon a level of danger it may represent to vehicle 100. For example, ifthe obstacle is a relatively shallow pothole, obstacle categorizer mayassign it to a low-risk category. If the obstacle is a patch of ice,obstacle categorizer may assign it to a medium-risk category. If theobstacle is a large slab of concrete, obstacle categorizer may assign itto a high-risk category. It will be appreciated that the examplesprovided herein for obstacles and their associated categories are notlimiting; the embodiments described herein may support othercombinations of obstacles and categories.

Obstacle categorizer 464 may also be operative to categorize obstaclesbased on their expected persistence. For example, a pothole may be apersistent obstacle; it may be expected to remain on roadway 20 forweeks, months, or even years after first being observed. Debris from atraffic accident may be a temporary obstacle; it may be expected to becleared from roadway 20 within hours or perhaps a day or two after firstbeing observed. Some obstacles may be recurring in nature; suchobstacles may be observed intermittently in the same location. Forexample, puddles or ice patches may tend to recur in the same locationsbased on the topography of roadway 20. Some obstacles may have hybridpersistence characteristics. For example, a pothole may also tend tofill with rain/slow and freeze during the winter. The pothole itself maybe of a persistent nature, whereas the resulting ice patch (or puddle)may be of a recurring nature. Obstacle categorizer 464 may categorize(step 535) the obstacle accordingly, e.g. as persistent, temporary,recurring, or hybrid.

Obstacle avoidance manager 460 may invoke obstacle policy manager 466 toset (step 539), based at least on the obstacle's category, anappropriate driving policy for avoiding (or at least minimizing therisks associated with) the obstacle.

If, per step 530, the obstacle is not anew obstacle, obstacle avoidancemanager 460 may use obstacle information stored in obstacle database 470to determine whether the current obstacle information is consistent withpreviously observed obstacle information (step 540). For example, apothole may expand in size and/or fill with water/ice compared to aprevious observation. Debris from a traffic accident may shift or bepartially removed. If the current obstacle information is inconsistentwith the stored obstacle information, obstacle avoidance manager 460 mayinvoke obstacle categorizer 464 to update (step 545) the category orcategories assigned in step 535, and obstacle policy manager 466 toupdate (step 549) the driving policy set in step 539.

As described hereinabove, even though a given obstacle report may beassociated with a previously issued obstacle warning, in some cases theobstacle report may not include sensor data indicative of an observationof the associated obstacle. Vehicle 100 may receive an obstacle warning,but by the time vehicle 100 reaches the location indicated in theobstacle warning, the associated obstacle may no longer be there. Forexample, based on obstacle information received by obstacle avoidanceserver 400, obstacle avoidance manager 460 may determine that debrisfrom traffic accident may be at a given location on roadway 20, and sendan obstacle warning to vehicle 100 for the given location. However, thedebris may be cleared by the time that vehicle 100 reaches the givenlocation. As described with reference to process 300, in such a case,vehicle 100 may provide an obstacle report to server 400 even though itdid not actually detect the obstacle associated with the obstaclewarning.

Accordingly, obstacle avoidance manager 460 may determine if an obstacleassociated with the obstacle report was actually detected by autonomousdriving system 200 or obstacle extractor 462 (step 550). If an obstaclewas detected per step 550, obstacle avoidance manager 460 may useobstacle timer 468 to set (step 555) a timer for the detected obstacle.The timer may be set according to an anticipated duration for theobstacle. For example, if the obstacle is debris from a trafficaccident, the timer may be set to a number of hours; for a pothole thetimer may be set for a number of weeks or months.

It will be appreciated that the detected obstacle from step 550 mayalready have had a timer set in a previous iteration of process 500,e.g., the obstacle report is associated with a known obstacle for whichan obstacle warning was previously generated and sent to vehicle(s) 100.In such a case, the timer may be reset to extend the anticipatedduration of the obstacle per a more recent observation. For example, ifthe obstacle was debris from a traffic accident, the timer may haveoriginally been set for two hours. If an obstacle report is receivedninety minutes later indicating that the debris is still there, obstacleavoidance manager 460 may reset the timer to update the anticipatedduration based on the latest information (i.e., the recent obstaclereport).

If an obstacle associated with the obstacle report was not detected byautonomous driving system 200 or obstacle extractor 462 (step 550),obstacle avoidance manager 460 may determine if the obstacle warning hasexpired (step 560) according to the timer (re)set in step 555. If thetimer has expired, obstacle avoidance manager 460 may remove (step 465)the obstacle from a warning list of currently relevant obstaclewarnings. Otherwise, processing control may continue to step 570.

Obstacle avoidance manager 460 may then update (step 570) the warninglist as necessary e.g., adding obstacle warnings, modifying drivingpolicies, removing obstacle warnings, etc. Obstacle avoidance manager460 may then send (step 580) the obstacle warning(s) from the warninglist to vehicle(s) 100. Depending on the configuration of systems 10,200, and 400, step 580 may be performed in real time or near real timeafter step 570. Alternatively, or in addition, step 580 may be performedin a periodic and/or event triggered batch process.

Process control may then return to step 510. It will be appreciated thatin some embodiments, steps 520-570 (or combinations, thereof) may beperformed iteratively, e.g., if multiple obstacles are extracted fromraw driving data.

Obstacle Detection Based on Visual and Non-Visual Information

Obstacles may be viewed as elements that introduce changes in thebehavior of a vehicle. Obstacles may be automatically detected byprocessing images obtained while a vehicle performs maneuvers that aresuspected as being obstacle avoidance maneuvers.

FIG. 6 illustrates a method 1600 executed by computerized system fordetecting obstacles. The computerized system may be located outside avehicle, may be distributed between different vehicles, and the like.

Method 1600 may start by step 1610 of receiving, from a plurality ofvehicles, and by an I/O module of a computerized system, visualinformation acquired during executions of vehicle maneuvers that aresuspected as being obstacle avoidance maneuvers.

Step 1610 may also include receiving behavioral information regardingbehavior of the plurality of vehicles, during an execution of themaneuvers that are suspected as being obstacle avoidance maneuvers. Thebehavioral information may represent the behavior (speed, direction ofpropagation, and the like) of the entire vehicle and/or may representthe behavior of parts or components of the vehicle (for example dampinga shock by a shock absorber, slowing the vehicle by the breaks, turns ofthe steering wheel, and the like), and/or may represent the behavior ofthe driver (the manner in which the driver drives), and the like.

A determination of what constitutes a vehicle maneuver suspected asbeing an obstacle avoidance maneuver may be detected in a supervisedmanner or in a unsupervised manner. For example—when in supervisedmanner method 1600 may include receiving examples of maneuvers that aretagged as obstacle avoidance maneuvers.

Such a maneuver may be an uncommon maneuver in comparison to at leastone out of (a) maneuvers of the same vehicle, (b) maneuvers performed atthe same location by other drivers, (c) maneuvers performed in otherplaces by other drivers, and the like. The maneuver may be compared toother maneuvers at the same time of day, at other times of day, at thesame date, at other dates and the like.

The vehicle maneuver suspected as being an obstacle avoidance maneuvermay be detected based on behavioral information that may be obtainedfrom one or more sensors such as an accelerometer, a wheel speed sensor,a vehicle speed sensor, a brake sensor, a steering wheel sensor, a shockabsorber sensor, an engine sensor, a driver sensor for sensing one ormore physiological parameter of the driver (such as heart beat), or anyother sensor—especially any other non-visual sensor.

The vehicle maneuver suspected as being an obstacle avoidance maneuvermay involve a change (especially a rapid change) in the direction and/orspeed and/or acceleration of the vehicle, a deviation from a lane, adeviation from a previous driving pattern followed at attempt to correctthe deviation, and the like.

For example—when applying supervised learning of obstacles—thecomputerized system may be fed with examples of vehicle behaviors whenbypassing known obstacles, when approaching a significant bump, and thelike.

For example—the determining may include ruling out common behaviors suchas stopping in front of a red traffic light.

For example—a fast and significant change in the speed and direction ofa vehicle at a linear road path may indicate that the vehicleencountered an obstacle.

The visual information may be raw image data (such as images) acquiredby a visual sensor of the vehicle, processed images, metadata or anyother type of visual information that represents the images acquired byvisual sensor.

The visual information may include one or more robust signatures of oneor more images acquired by one or more visual sensors of the vehicle. Anon-limiting example of a robust signature is illustrated in U.S. Pat.No. 8,326,775 which is incorporated herein by reference.

Step 1610 may be followed by step 1620 of determining, based at least onthe visual information, at least one visual obstacle identifier forvisually identifying at least one obstacle.

The determining may include clustering visual information based on theobjects represented by the visual information, and generating a visualobstacle identifier that represents a cluster. The clusters may belinked to each other and a visual obstacle identifier may represent aset of linked clusters.

The determining may include filtering out objects that are representedin the visual information based on a frequency of appearance of theobjects in the visual information. For example—the road may appear invirtually all images and may be ignored.

The clustering may or may not be responsive to the behavior of thevehicle during the executions of vehicle maneuvers that are suspected asbeing obstacle avoidance maneuvers.

The visual obstacle identifier may be a model of the obstacle, a robustsignature of the obstacle, configuration information of a neural networkrelated to the obstacle.

A model of the obstacle may define one or more parameters of theobstacle such as shape, size, color of pixels, grayscale of pixels, andthe like.

The configuration information of a neural network may include weights tobe assigned to a neural network. Different neural networks may be usedto detect different obstacles.

The visual obstacle identifier for visually identifying an obstacle mayidentify a group of obstacles to which the obstacle belongs. Forexample—the visual obstacle identifier may identify a concept. See, forexample, U.S. Pat. No. 8,266,185 which is incorporated herein byreference.

A visual obstacle identifier that identifies an obstacle may include atleast one out of severity metadata indicative of a severity of theobstacle, type metadata indicative of a type of the obstacle (forexample—pothole, tree, bump), size metadata indicative of a size of theobstacle, timing metadata indicative of a timing of an existence of theobstacle (for example—persistent, temporary, recurring), and locationmetadata indicative of a location of the obstacle. See, for exampleseverity metadata 1851, type metadata 1852, size metadata 1853, timingmetadata 1854 and location metadata 1855 of FIG. 7.

Step 1620 may be followed by step 1630 of responding to the outcome ofstep 1620. For example—step 1630 may include transmitting to one or moreof the plurality of vehicles, the at least one visual obstacleidentifier. Additionally or alternatively, step 1630 may includepopulating an obstacle database. The visual obstacle identifier may beincluded in an obstacle warning.

The behavioral information is obtained by non-visual sensors of theplurality of vehicles.

Step 1630 may also include verifying, based on at least the visualinformation, whether the maneuvers that are suspected as being obstacleavoidance maneuvers are actually obstacle avoidance maneuvers.

The verification may include detecting at least a predefined number ofindications about a certain obstacle.

The verification may include receiving information from another source(for example input from drivers, information provided from the police,department of transportation or other entity responsible to themaintenance of roads, regarding the existence of obstacles.

Step 1630 may be followed by step 1640 of transmitting to one or more ofthe plurality of vehicles, verification information indicative of themaneuvers that are actually obstacle avoidance maneuvers. This may helpthe vehicles to ignore maneuvers that were suspected as obstacleavoidance maneuvers—which are not actual obstacle avoidance maneuvers.

Method 1600 enables a vehicle that receives a visual obstacle identifierto identify an obstacle of known parameters—even if the vehicle is thefirst one to image that obstacle. Method 1600 may be executed withouthuman intervention and may detect obstacles without prior knowledge ofsuch obstacles. This provides a flexible and adaptive method fordetecting obstacles. The visual information and/or the visual obstacleidentifier may be very compact—thereby allowing the learning ofobstacles and afterwards detecting obstacles to be executed using arelatively small amount of computational and/or storage and/orcommunication resources.

It should be noted that different vehicle may perform obstacle avoidancemaneuvers that may differ from each other by duration, vehicle behaviorand the like. An example is illustrated in FIGS. 8-10 in which first,second and third vehicles perform different obstacle avoidance maneuversand acquire different numbers of images during said obstacle avoidancemaneuvers.

FIG. 8 illustrates a first vehicle (VH1) 1801 that propagates along aroad 1820. First vehicle 1801 performs a maneuver 1832 suspected asbeing a obstacle avoidance maneuver when encountered with obstacle 1841.Maneuver 1832 is preceded by a non-suspected maneuver 1831 and isfollowed by another non-suspected maneuver 1833.

First vehicle 1801 acquires a first plurality (N1) of imagesI1(1)-I1(N1) 1700(1,1)-1700(1,N1) during obstacle avoidance maneuver1832.

Visual information V1(1)-V1(N1) 1702(1,1)-1702(1,N1) is sent from firstvehicle 1801 to computerized system (CS) 1710 via network 1720.

The visual information may be the images themselves. Additionally oralternatively, first vehicle processes the images to provide arepresentation of the images.

First vehicle 1801 may also transmit behavioral information B1(1)-B1(N1)1704(1,1)-1704(1,N1) that represents the behavior of the vehicle duringmaneuver 1832.

FIG. 9 illustrates a second vehicle (VH2) 1802 that propagates along aroad 1820. Second vehicle 1802 performs a maneuver 1833 suspected asbeing a obstacle avoidance maneuver when encountered with obstacle 1841.Maneuver 1832 is preceded by a non-suspected maneuver and is followed byanother non-suspected maneuver.

Second vehicle 1802 acquires a second plurality (N2) of imagesI2(1)-I2(N2) 1700(2,1)-1700(2,N2) during maneuver 1833.

Visual information V2(1)-V2(N2) 1702(2,1)-1702(2,N2) is sent from secondvehicle 1802 to computerized system (CS) 1710 via network 1720.

The visual information may be the images themselves. Additionally oralternatively, second vehicle processes the images to provide arepresentation of the images.

Second vehicle 1802 may also transmit behavioral information (not shown)that represents the behavior of the vehicle during maneuver 1832.

FIG. 10 illustrates a third vehicle (VH2) 1803 that propagates along aroad. Third vehicle 1803 performs a maneuver 1834 suspected as being anobstacle avoidance maneuver when encountered with obstacle 1841.Maneuver 1834 is preceded by a non-suspected maneuver and is followed byanother non-suspected maneuver.

Third vehicle 1803 acquires a third plurality (N3) of imagesI3(1)-I3(N3) 1700(3,1)-1700(3,N3) during maneuver 1834.

Visual information V3(1)-V3(N3) 1702(3,1)-1702(3,N3) is sent from thirdvehicle 1803 to computerized system (CS) 1710 via network 1720.

The visual information may be the images themselves. Additionally oralternatively, third vehicle processes the images to provide arepresentation of the images.

Third vehicle 1803 may also transmit behavioral information (not shown)that represents the behavior of the vehicle during maneuver 1832.

FIG. 11 illustrates a method 1900 for detecting obstacles. The methodmay be executed by a vehicle.

Method 1900 may start by steps 1910 and 1920.

Step 1910 may include sensing, by a non-visual sensor of a vehicle, abehavior of a vehicle. The non-visual sensor may be an accelerometer, ashock absorber sensor, a brakes sensor, and the like.

Step 1920 may include acquiring, by a visual sensor of the vehicle,images of an environment of the vehicle. Step 1920 may continue evenwhen such a suspected maneuver is not detected.

Steps 1920 and 1930 may be followed by step 1930 of determining, by aprocessing circuitry of the vehicle, whether the behavior of the vehicleis indicative of a vehicle maneuver that is suspected as being anobstacle avoidance maneuver.

The vehicle maneuver suspected as being an obstacle avoidance maneuvermay be detected based on behavioral information that may be obtainedfrom one or more sensors such as an accelerometer, a wheel speed sensor,a vehicle speed sensor, a brake sensor, a steering wheel sensor, anengine sensor, a shock absorber sensor, a driver sensor for sensing oneor more physiological parameter of the driver (such as heart beat), orany other sensor—especially any other non-visual sensor.

The vehicle maneuver suspected as being an obstacle avoidance maneuvermay involve a change (especially a rapid change) in the direction and/orspeed and/or acceleration of the vehicle, a deviation from a lane, adeviation from a previous driving pattern followed at attempt to correctthe deviation, and the like.

Step 1930 may be followed by step 1940 of processing the images of theenvironment of the vehicle obtained during a vehicle maneuver that issuspected as being the obstacle avoidance maneuver to provide visualinformation.

The visual information may be raw image data (such as images) acquiredby a visual sensor of the vehicle, processed images, metadata or anyother type of visual information that represents the images acquired byvisual sensor.

The visual information may include one or more robust signatures of oneor more images acquired by one or more visual sensors of the vehicle. Anon-limiting example of a robust signature is illustrated in U.S. Pat.No. 8,326,775 which is incorporated herein by reference.

Step 1940 may be followed by step 1950 of transmitting the visualinformation to a system that is located outside the vehicle.

Method 1900 may also include step 1960 of receiving at least one visualobstacle identifier from a remote computer.

Method 1900 may include step 1970 receiving verification informationindicative of whether the maneuver that was suspected being an obstacleavoidance maneuver is actually an obstacle avoidance maneuver. This maybe used in the vehicle during step 1930.

Methods 1900 and 1600 illustrate a learning process. The products of thelearning process may be used to detect obstacles—during an obstacledetection process that uses the outputs of the learning process.

The obstacle detection process may include detecting newly detectedobstacles that were not detected in the past.

The learning process may continue during the obstacle detection processor may terminate before the obstacle detection process.

FIG. 12 illustrates method 2100 for detecting an obstacle.

Method 2100 may start by step 2110 of receiving, by an I/O module of avehicle, a visual obstacle identifier for visually identifying anobstacle; wherein the visual obstacle identifier is generated based onvisual information acquired by at least one visual sensor during anexecution of at least vehicle maneuver that is suspected as being anobstacle avoidance maneuver. Thus step 2110 may include receivingoutputs of the learning processes of method 1600.

Method 2100 may also include step 2120 of acquiring, by a visual sensorof the vehicle, images of an environment of the vehicle.

Step 2120 may be followed by step 2130 of searching, by a processingcircuitry of the vehicle, in the images of the environment of thevehicle for an obstacle that is identified by the visual obstacleidentifier.

If finding such an obstacle then step 2130 may be followed by step 2140of responding, by the vehicle, to a detection of an obstacle.

The responding may be based on a driving policy, may be responsive tothe mode of controlling the vehicle (any level of automation—startingfrom fully manual control, following by partial human intervention andending with fully autonomous).

For example—the responding may include generating an alert perceivableby a human driver of the vehicle, sending an alert to a computerizedsystem located outside the vehicle, handing over the control of thevehicle to a human driver, handing over a control of the vehicle to anautonomous driving manager, sending an alert to an autonomous drivingmanager, performing an obstacle avoidance maneuver, determining whetherthe obstacle is a newly detected obstacle, informing another vehicleabout the obstacle, and the like.

Determining a Location of a Vehicle

A vehicle, especially but not limited to an autonomous vehicle, has toknown its exact location in order to propagate in an efficient manner.For example—the vehicle may receive an obstacle warning that includesthe location of the obstacle.

Responding to the obstacle warning may require to know the exactlocation of the vehicle. Furthermore—an autonomous driving operation mayrequire a knowledge of the exact location of the vehicle or may beexecuted in a more optimal manner if the exact location of the vehicleis known. In general, the processing burden associated with autonomousdriving decision may be lowered when an autonomous driving manager isaware of the location of the vehicle.

Global positioning systems (GPS) are inaccurate and cannot be solelyused for locating the exact location of a vehicle.

There is a growing need to determine the exact location of the vehiclein an efficient manner.

There is a provided an efficient and accurate method for determining theexact location of a vehicle.

The method is image-based in the sense that it is based on imagesacquired by the vehicle and on a comparison between the acquired imagesand reference information that was acquired at predefined locations.

The determining of the actual location of the vehicle is of a resolutionthat is smaller than a distance between adjacent reference images. Forexample—the resolution may be a fraction (for example between 1-15%,between 20-45%, of between 5-50%) of the distance between adjacentreference images. In the following example it is assumed that thedistance between adjacent reference images is about 1 meter and theresolution is about 10 to 40 centimeters.

FIG. 13 illustrates method 1300 for determining a location of a vehiclereceiving reference visual information that represents multiplereference images acquired at predefined locations.

Method 1300 may start by step 1310 of receiving reference visualinformation that represents multiple reference images acquired atpredefined locations.

The multiple reference images may be taken within an area in which thevehicle is located. The area may be of any size and/or dimensions. Forexample, the area may include a part of a neighborhood, a neighborhood,a part of a city, a city, a part of a state, a state, a part of acountry, a country, a part of a continent, a continent, and even theentire world.

A computerized system and the vehicle may exchange reference visualinformation in real-time (or near real-time) and/or in a burst mode,where the reference visual information may be provided eitherperiodically and/or when conditions are suitable for a data exchangebetween the computerized system and the vehicle. For example, a vehiclemay be configured to perform regular uploads/downloads of referencevisual information when the engine is turned on (or off, with a keepalive battery function to facilitate communication with the computerizedsystem). Alternatively, or in addition, the vehicle may be configured toperform such uploads/downloads when stationary (e.g., parked, or waitingat a traffic light) and a sufficiently reliable wireless connection isdetected. Alternatively, or in addition, the uploads/downloads may betriggered by location, where the reference visual information may bedownloaded from the computerized system when the vehicle enters an areafor which it does not have up-to-date reference visual information.

The reference visual information may include the reference imagesthemselves or any other representation of the reference images.

The reference information may include robust signatures of referenceimages. The robust signature may be robust to at least one out of noise,lighting conditions, rotation, orientation, and the like—which mayprovide a robust location detection signatures. Furthermore—using robustsignatures may simplify the acquisition of the reference images—as thereis no need to acquired reference images in different lighting conditionsor other conditions to which the robust signature is indifferent.

The comparison between the reference information and information fromthe acquired image may be even more efficient when the reference visualinformation and information regarding the acquired image is representedin a cortex representation.

A cortex representation of an image is a compressed representation ofthe visual information that is generated by applying a cortex functionthat include multiple compression iteration. A signature of an image maybe a map of firing neurons that fire when a neural network is fed withan image. The map undergoes multiple compression iterations during whichpopular strings are replaced by shorter representations.

Method 1300 may also include step 1320 of acquiring, by a visual sensorof the vehicle, an acquired image of an environment of the vehicle. Itshould be noted that multiple images may be acquired by the vehicle—butfor simplicity of explanation the following text will refer to a singleimage. The vehicle may repetitively determine its location—and multiplerepetitions of steps 1320, 1330 and 1340 may be repeated multipletimes—for different images.

Step 1330 may include generating, based on the acquired image, acquiredvisual information related to the acquired image.

The location determination may be based on visual image regarding staticobstacles. Accordingly—each one of the reference visual information andthe acquired visual information may include static visual informationrelated to static objects.

The extraction of the static visual information may include reducingfrom the visual information any visual information that is related todynamic objects. The extraction of the static information may or may notrequire object recognition.

The acquired visual information may include acquired static visualinformation that may be related to at least one static object locatedwithin the environment of the vehicle.

Step 1330 may be followed by step 1340 of searching for a selectedreference image out of the multiple reference images. The selectedreference image may include selected reference static visual informationthat best matches the acquired static visual information.

Step 1340 may be followed by step 1350 of determining an actual locationof the vehicle based on a predefined location of the selected referenceimage and to a relationship between the acquired static visualinformation and to the selected reference static visual information.

The determining of the actual location of the vehicle may be of aresolution that may be smaller than a distance between the selectedreference image and a reference image that may be immediately followedby the selected reference image.

The determining of the actual location of the vehicle may includecalculating a distance between the predefined location of the selectedreference image and the actual location of the vehicle based on arelationship between at least one value of a size parameter of the atleast one static object within the selected reference image and at leastone value of the size parameter of the at least one static object withinthe acquired image.

The determining of the actual location of the vehicle may includedetermining whether the actual location of the vehicle precedes thepredefined location of the selected reference image or precedes thepredefined location of the selected reference image.

The method may include subtracting the first distance from thepredefined location of the selected reference image when determiningthat the actual location of the vehicle precedes the predefined locationof the selected reference image.

The method may include adding the first distance from the predefinedlocation of the selected reference image when determining that theactual location of the vehicle follows the predefined location of theselected reference image.

Each predefined location may be associated with a single referenceimage.

Alternatively, each reference image may be associated with multipleimages that differ from each other by scale. The images that differ fromeach other by scale may represent different relative distances betweenthe vehicle and the predefined image—larger scale may be related tosmaller distances.

Using images of different scale (or visual information related to imagesof different scale) may reduce the computational load associated withlocation determination—as it may eliminate the need to manipulate asingle reference image to find an exact math that may determine thedistance between the vehicle and the reference location.

Using visual information related to images of different scale may alsoassist when the the visual information is a lossy representation of theimages in which a representation of an object of certain scale cannot beeasily generated based on a representation of the same object of anotherscale. The visual information may be a signature of an image may be amap of firing neurons that fire when a neural network is fed with animage. See, for example, the signature of U.S. Pat. Nos. 8,326,775 and8,312,031, assigned to common assignee, which are hereby incorporated byreference.

FIG. 14 illustrates an example of a vehicle VH1 1402 that acquires animage I(k) 1404 and tries to determine its location based on analysis ofvisual information it generates from image I(k) and multiple referenceimages RI(1,1)-RI(j,1) 1402(1)-1402(j) and RI(j+1,1)-RI(j)1402(j+1,1)-1402(J,1). The front of the vehicle is positioned betweenthe predefined locations associated with reference images RI(j) andRI(j+1)—and one of these images should be associated with referencevisual information that best matches the visual information associatedwith image I(k). The distance between consecutive predefined locationsis DRI 1422—and the location resolution is finer than DRI.

FIG. 15 illustrates an example of a vehicle VH1 1402 that acquires animage I(k) 1404 and tries to determine its location based on analysis ofvisual information (1406) it generates from image I(k) and visualinformation related to reference images. In FIG. 15 each predefinedlocation is associated with a plurality of reference images—with aplurality (p) of reference visual information per referencelocation—1408(1,1)-1408(j,1), 1408(j+1,1)-1408(J,1),1409(1,p)-1409(j,p), and 1409(j+1,p)-1409(J,p).

The front of the vehicle is positioned between the j'th and (j+1)'thpredefined locations and the vehicle may determine the best matchingpredefined location and also the distance from the predefined locationsbased on the p different reference visual information associated withone of these predefined locations.

FIGS. 17 and 18 illustrate an example of an image and static visualinformation.

FIG. 17 illustrates an example of an image acquired by a vehicle. Theimage shows two lanes 1411 and 1412 of a road, right building 1413,vehicles 1414 and 1415, first tree 1416, table 1417, second tree 1418,cloud 1419, rain 1420, left building 1421, shrubbery 1422, child 1423and ball 1424.

The dynamic objects of this image include vehicles 1414 and 1415, cloud1419, rain 1420, shrubbery 1422, child 1423 and ball 1424. These dynamicobjects may be represented by visual information (such as signatures)that describe them as dynamic—and visual information related to thesedynamic objects may be removed from the visual information of the imagesto provide static visual information.

FIG. 18 illustrates the removal (blocks 1432) of dynamic visualinformation from the visual information 1430 to provide static visualinformation 1431.

Visual information 1430 may be a map of firing neurons of a network thatfired when the neural network was fed with image I(k)- or may be acompressed representation (for example a cortex representation) of suchmap.

Triggering Human Intervention

FIG. 19 illustrates method 1450.

Method 1450 may be aimed to find at least one trigger for humanintervention in a control of a vehicle.

Method 1450 may start by step 1452 of receiving, from a plurality ofvehicles, and by an I/O module of a computerized system, visualinformation acquired during situations that are suspected as situationsthat require human intervention in the control of at least one of theplurality of vehicles.

Step 1452 may be followed by step 1456 of determining, based at least onthe visual information, the at least one trigger for human intervention.The at least one trigger is represented by trigger visual information.

Step 1456 may include at least one out of

-   -   Determining a complexity of the situation.    -   Determining a danger level associated with the situation.    -   Determining in response to statistics of maneuvers executed by        different vehicles during a same situation that is suspected as        a situation that requires human intervention in the control of        at least one of the plurality of vehicles.    -   Determining based on generated or received movement information        of entities included in the visual information. The movement        information may represent entity movement functions of the        entities.    -   The determining may include estimating, based on the entity        movement functions, a future movement of the entities.

The determining of step 1456 may be executed in an unsupervised manneror a supervised manner.

The determining of step 1456 is responsive to at least one humanintervention policy of the at least one vehicle. The human interventionpolicy of a vehicle may differ from the human intervention policy ofanother vehicle.

A human intervention policy of a vehicle may define certain criteriahuman intervention such as the danger level of a situation, thecomplexity of the situation (especially complexity of maneuvers requiredto overcome the situation), the potential damage that may result to thevehicle, driver or surroundings due to the situation. A humanintervention policy of a vehicle may also define certain situations thatrequire human intervention—such as specific locations (for example neara cross road of a school), or specific content (near a school bus), andthe like.

A trigger may be determined during step 1456 when the situation or oneof the attributes of the situation (danger level, location, combinationof location and time) match those of the human intervention policy.

Situations that are suspected as requiring human intervention in thecontrol of a vehicle may be viewed as situations that are complex and/ordangerous and/or require changes in the behavior of a vehicle. Thesesituations may result from an obstacle or may result from other reasons.

Situations that are suspected as requiring human intervention in thecontrol of a vehicle may be automatically detected by processing imagesobtained while a vehicle performs maneuvers that are suspected asresulting from situations that are suspected as requiring humanintervention in the control of a vehicle.

Situations that are suspected as requiring human intervention in thecontrol of a vehicle may be detected based on behavioral informationregarding behavior of the plurality of vehicles, during an execution ofthe maneuvers that are suspected as resulting from situations that aresuspected as requiring human intervention in the control of a vehicle.

The behavioral information may represent the behavior (speed, directionof propagation, and the like) of the entire vehicle and/or may representthe behavior of parts or components of the vehicle (for example dampinga shock by a shock absorber, slowing the vehicle by the breaks, turns ofthe steering wheel, and the like), and/or may represent the behavior ofthe driver (the manner in which the driver drives), and the like.

After a situation is tagged as requiring human intervention in thecontrol of a vehicle—the vehicle may trigger human intervention whenthis situation is visually detected. For example—some events (or sometypes of events) may be defined (even based on previous recognition of asituation as requiring human intervention in the control of a vehicle)as requiring human intervention. For example—an image of a cross roadlocated near a school, an image of a school bus that transportschildren, an image of a bar and drunken pedestrians near the bar, animage of multiple people that are near the road, and the like may beflagged as situation as requiring human intervention in the control of avehicle

A determination of what constitutes a vehicle maneuver suspected asrequiring human intervention in the control of a vehicle may be detectedin a supervised manner or in a unsupervised manner.

Such a maneuver may be an uncommon maneuver in comparison to at leastone out of (a) maneuvers of the same vehicle, (b) maneuvers performed atthe same location by other drivers, (c) maneuvers performed in otherplaces by other drivers, and the like. The maneuver may be compared toother maneuvers at the same time of day, at other times of day, at thesame date, at other dates and the like.

The vehicle maneuver suspected as related to a situation that requireshuman intervention in the control of a vehicle may involve a change(especially a rapid change) in the direction and/or speed and/oracceleration of the vehicle, a deviation from a lane, a deviation from aprevious driving pattern followed at attempt to correct the deviation,and the like.

For example—when applying supervised learning of obstacles—thecomputerized system may be fed with examples of vehicle behaviors thatrequire human intervention in the control of a vehicle.

For example—the determining may include ruling out common behaviors suchas stopping in front of a red traffic light.

For example—a fast and significant change in the speed and direction ofa vehicle at a linear road path may indicate that the vehicleencountered a situation requiring human intervention in the control of avehicle

The visual information may be raw image data (such as images) acquiredby a visual sensor of the vehicle, processed images, metadata or anyother type of visual information that represents the images acquired byvisual sensor.

The visual information may include one or more robust signatures of oneor more images acquired by one or more visual sensors of the vehicle. Anon-limiting example of a robust signature is illustrated in U.S. Pat.No. 8,326,775 which is incorporated herein by reference.

Step 1456 may be followed by step 1460 of transmitting to one or more ofthe plurality of vehicles, the trigger visual information.

FIG. 20 illustrates a method 1490 for detecting situation that requireshuman intervention in the control of a vehicle. The method may beexecuted by a vehicle.

Method 1490 may start by steps 1491 and 1492.

Step 1491 may include sensing, by a non-visual sensor of a vehicle, abehavior of a vehicle. The non-visual sensor may be an accelerometer, ashock absorber sensor, a brakes sensor, and the like.

Step 1492 may include acquiring, by a visual sensor of the vehicle,images of an environment of the vehicle.

Steps 1491 and 1492 may be followed by step 1493 of determining, by aprocessing circuitry of the vehicle, whether the behavior of the vehicleis indicative of a vehicle maneuver that is related to a situation thatis suspected as requiring human intervention in the control of avehicle.

The vehicle maneuver related to a situation that is suspected asrequiring human intervention in the control of a vehicle may be detectedbased on behavioral information that may be obtained from one or moresensors such as an accelerometer, a wheel speed sensor, a vehicle speedsensor, a brake sensor, a steering wheel sensor, an engine sensor, ashock absorber sensor, a driver sensor for sensing one or morephysiological parameter of the driver (such as heart beat), or any othersensor—especially any other non-visual sensor.

The vehicle maneuver related to a situation that is suspected asrequiring human intervention in the control of a vehicle may involve achange (especially a rapid change) in the direction and/or speed and/oracceleration of the vehicle, a deviation from a lane, a deviation from aprevious driving pattern followed at attempt to correct the deviation,and the like.

Step 1493 may be followed by step 1494 of processing the images of theenvironment of the vehicle obtained during a vehicle maneuver related toa situation that is suspected as requiring human intervention in thecontrol of a vehicle to provide visual information.

The visual information may be raw image data (such as images) acquiredby a visual sensor of the vehicle, processed images, metadata or anyother type of visual information that represents the images acquired byvisual sensor.

The visual information may include one or more robust signatures of oneor more images acquired by one or more visual sensors of the vehicle. Anon-limiting example of a robust signature is illustrated in U.S. Pat.No. 8,326,775 which is incorporated herein by reference.

Step 1494 may be followed by step 1495 of transmitting the visualinformation to a system that is located outside the vehicle.

Method 1490 may also include step 1496 of receiving at least one visualobstacle identifier from a remote computer.

Method 1490 may include step 1497 receiving verification informationindicative of whether the maneuver related to a situation that issuspected as requiring human intervention in the control of a vehiclemaneuver is actually related to a situation that actually requires humanintervention in the control of a vehicle. This may be used in thevehicle during step 1493.

Methods 1450 and 1490 illustrate a learning process. The products of thelearning process may be used to detect situations that require humanintervention in the control of a vehicle—during an vehicle controlprocess that uses the outputs of the learning process.

The learning process may continue during the vehicle control process ormay terminate before the vehicle control process.

FIG. 21 illustrates method 1470.

Method 1470 may start by step 1472 of receiving, by an I/O module of avehicle, trigger visual information for visually identifying situationsthat require human intervention in the control of a vehicle. The visualobstacle identifier is generated based on visual information acquired byat least one visual sensor during an execution of at least vehiclemaneuver that is suspected as being an obstacle avoidance maneuver. Thusstep 1472 may include receiving outputs of the learning processes ofmethod 1450.

Method 1470 may also include step 1474 of acquiring, by a visual sensorof the vehicle, images of an environment of the vehicle.

Step 1474 may be followed by step 1476 of searching, by a processingcircuitry of the vehicle, images of the environment of the vehicle for asituation that is identified by the trigger visual information.

If finding such a situation then step 1476 may be followed by step 1478of triggering human intervention in the control of a vehicle. This mayinclude alerting the driver that human intervention is required, handingthe control of the vehicle to the driver- and the like.

Step 1478 may include, for example—generating an alert perceivable by ahuman driver of the vehicle, sending an alert to a computerized systemlocated outside the vehicle, handing over the control of the vehicle toa human driver, handing over a control of the vehicle from an autonomousdriving manager, sending an alert to an autonomous driving manager,informing another vehicle about the trigger, and the like.

FIGS. 22-27 illustrate example of situations that are suspected asrequiring human intervention in the control of a vehicle.

FIG. 22 illustrates first vehicle VH1 1801 as stopping (position 1501)in front of a puddle 1506 and then passing the puddle (may drivestraight or change its direction till ending the maneuver at point 1502.The maneuver may be indicative that passing the puddle may require humanintervention.

Visual information acquired between positions 1501 and 1502 areprocessed during step 1494.

FIG. 23 illustrates first vehicle VH1 1801 as sensing pedestrians 1511and 1512. These pedestrians either are associated with temporalsignatures that illustrate their movement (1511′ and 1512′) and/or thevehicle may sense the movements of the pedestrians—and the vehicle maysend (based on the movement) the future movement and the future location(1511″ and 1512″) of the pedestrians and perform a maneuver that mayinclude altering speed (for example stopping at point 1513) and/orbypassing the pedestrians (maneuver 1515).

Visual information acquired between positions 1513 and 1514 (end of themaneuver) are processed during step 1494.

FIG. 24 illustrates first vehicle VH1 1801 as sensing parked vehiclesPV1 1518 and PV2 1519 that part on both sides of a double-lanebi-directional road, that require the first vehicle to perform a complexmaneuver 1520 that includes changing lanes and changing directionrelatively rapidly.

Visual information acquired between positions 1516 (beginning of themaneuver) and 1517 (end of the maneuver) are processed during step 1494.

FIG. 25 illustrates first vehicle VH1 1801 as stopping (position 1522)in front of wet segment of the road on which rain 1521 (from cloud 1522)falls. The stop (at location 1522) and any further movement after movingto another part of the road may be regarded as a maneuver 1523 that isindicative that passing the wet segment may require human intervention.

Visual information acquired between position 1522 (beginning of themaneuver) and the end of the maneuver are processed during step 1494.

FIG. 26 illustrates first vehicle VH1 1801 as stopping (position 1534)in front of a situation that may be labeled as a packing or unpackingsituation—a track 1531 is parked on the road, there is an open door1532, and a pedestrian 1533 carries luggage on the road. The firstvehicle 1801 bypasses the truck and the pedestrian between locations1534 and 1535 during maneuver 1539. The maneuver may be indicative thata packing or unpacking situation may require human intervention.

Visual information acquired between positions 1534 and 1535 areprocessed during step 1494.

FIG. 27 illustrates first vehicle VH1 1801 as turning away (maneuver1540) from the road when sensing that it faces a second vehicle VH2 1802that moves towards VH1 1801. Maneuver 1540 may be indicative that suchas potential collision situation may require human intervention.

Visual information acquired between positions 1541 and 1542 (start andstop of maneuver 1540) are processed during step 1494.

Tracking after an Entity

There may be beneficial to detect and object in a robust and anefficient manner by taking into account the temporal behavior of theentity and also its spatial signature. Using signatures of multiple typemay increase the reliability of the detection, one signature may verifythe other signature. In addition—one signature may assist in detectingthe entity when the other signature is missing or of a low quality.Furthermore—estimating the future movement of an entity—based on thespatial signature of the entity (or of those of entities of the sametype) may enable to predict the future effect that this entity may haveon the vehicle and allow to determine the future driving pattern of thevehicle in advance.

The temporal signature may be compresses—thus saving memory and allowinga vehicle to allocate limited memory resources for tracking eachentity—even when tracking multiple entities.

FIG. 28 illustrates a method 2600 for tracking after an entity.

Method 2600 may include steps 2602, 2604, 2606 and 2608.

Tracking, by a monitor of a vehicle, a movement of an entity thatappears in various images acquired during a tracking period. S2602.

Generating, by a processing circuitry of the vehicle, an entity movementfunction that represents the movement of the entity during the trackingperiod. S2604.

Generating, by the processing circuitry of the vehicle, a compressedrepresentation of the entity movement function. S2606.

Responding to the compressed representation of the entity movementfunction. S2608.

The compressed representation of the entity movement function may beindicative of multiple properties of extremum points of the entitymovement function.

The multiple properties of an extremum point of the extremum points, mayinclude a location of the extremum point, and at least one derivative ofthe extremum point.

The multiple properties of an extremum point of the extremum points mayinclude a location of the extremum point, and at least two derivative ofat least two different orders of the extremum point.

The multiple properties of an extremum point of the extremum points, mayinclude a curvature of the function at a vicinity of the extremum point.

The multiple properties of an extremum point of the extremum points, mayinclude a location and a curvature of the function at a vicinity of theextremum point.

Method 2600 may include acquiring the images by a visual sensor of thevehicle. S2601.

At least one image of the various images may be acquired by an imagesensor of another vehicle.

Step 2608 may include at least one out of:

-   -   Storing, in a memory unit of the vehicle, the compressed        representation of the entity movement function.    -   Transmitting the compressed representation of the entity        movement function to a system that may be located outside the        vehicle.    -   Estimating, by a processing circuitry of the vehicle, a future        movement of the entity, based on the compressed representation        of the entity movement function.    -   Generating a profile of the entity, by a processing circuitry of        the vehicle, based on the compressed representation of the        entity movement function.    -   Predicting an effect of a future movement of the entity on a        future movement of the vehicle, wherein the predicting may be        executed by a processing circuitry of the vehicle, and may be        based on the compressed representation of the entity movement        function.    -   searching for a certain movement pattern within the movement of        the entity, by a processing circuitry of the vehicle, based on        the compressed representation of the entity movement function.

Method 2600 may include receiving a compressed representation of anotherentity movement function, the other entity movement function may begenerated by another vehicle and may be indicative of the movement ofthe entity during at least a subperiod of the tracking period. S2610.

Method 2600 may include amending the compressed representation of theentity movement function based on the compressed representation of theother entity movement function. S2612.

Method 2600 may include determining a duration of the tracking period.S2603. The duration may be determine based on the certainty level of theprediction, based on memory and/or computational resources allocated tothe tracking, and the like.

FIG. 29 illustrates examples of entity movement functions. One axis istime. Other axes are spatial axes (x-axis, y-axis and the like).

FIG. 30 illustrates method 2630.

Method 2630 may include steps 2631, 2632, 2634 and 2636.

Method 2630 may include calculating or receiving an entity movementfunction that represents a movement of the entity during a trackingperiod. S2632.

Searching, by a search engine, for a matching reference entity movementfunction. S2634

Identifying the entity using reference identification information thatidentifies a reference entity that exhibits the matching referenceentity movement function. S2636.

The reference identification information may be a signature of theentity.

The method may include acquiring a sequence of images by an imagesensor; and calculating the entity movement based on the sequence ofimages. S2631

FIG. 31 illustrates a method 2640.

Method 2630 may include steps 2642, 2644, 2646 and 2648.

Method 2640 may start by calculating or receiving multiple entitymovement functions that represent movements of multiple entities. S2642.

Clustering the multiple entity movement functions to clusters. S2644.

For each cluster, searching, by a search engine, for a matching type ofreference entity movement functions. S2646.

Identifying, for each cluster, a type of entity, using referenceidentification information that identifies a type of reference entitiesthat exhibits the matching type of reference entity movement functions.S2648.

FIG. 32 illustrates a method 2650.

Method 2650 may include steps 2652, 2654 and 2656.

Method 2650 may start by calculating or receiving (a) an entity movementfunction that represents a movement of an entity, and (b) a visualsignature of the entity. S2652.

Comparing the entity movement function and the visual signature toreference entity movement functions and reference visual signatures ofmultiple reference objects to provide comparison results. S2654.

Classifying the object as one of the reference objects, based on thecomparison results. S2656.

FIG. 33 illustrates a method 2660.

Method 2660 may include steps 2662, 2664, 2666 and 2668.

Step 2662 may include calculating or receiving an entity movementfunction that represents a movement of an entity.

Step 2664 may include comparing the entity movement function toreference entity movement functions to provide comparison results.

Step 2666 may include classifying the object as a selected referenceobject of the reference objects, based on the comparison results.

Step 2668 may include verifying the classifying of the object as theselected reference object by comparing a visual signature of the objectto a reference visual signature of the reference object.

FIG. 34 illustrates a method 2670.

Method 2670 may include steps 2672, 2674, 2676 and 2678.

Step 2672 may include calculating or receiving a visual signature of theobject.

Step 2674 may include comparing a visual signature of the object toreference visual signatures of multiple reference objects to providecomparison results.

Step 2644 may include classifying the object as a selected referenceobject of the reference objects, based on the comparison results.

Step 2678 may include verifying the classifying of the object as theselected reference object by comparing an entity movement function thatrepresents a movement of the entity to a reference entity movementfunctions to provide comparison results.

FIG. 35 illustrates a method 2680.

Method 2680 may include steps 2682, 2684 and 2686.

Method 2680 is for generating a signature of an object.

Step 2682 may include calculating or receiving a visual signature of theobject.

Step 2684 may include calculating or receiving an entity movementfunction that represents a movement of the object.

Step 2686 may include generating a spatial-temporal signature of theobject that represents the visual signature and the entity movementfunction of the object.

FIG. 36 illustrates a method 2700.

Method 2700 may include steps 2702, 2704 and 2706.

Method 2700 is for driving a first vehicle based on information receivedfrom a second vehicle.

The exchange of information—especially compact and robust signaturesbetween vehicles may improve the driving of the vehicle and at a modestcost—especially when the vehicles exchange compact signatures betweenthemselves.

Step 2702 may include receiving, by the first vehicle, acquired imageinformation regarding (a) a signature of an acquired image that wasacquired by the second vehicle, (b) a location of acquisition of theacquired image.

Step 2704 may include extracting, from the acquired image information,information about objects within the acquired image.

Step 2706 may include preforming a driving related operation of thefirst vehicle based on the information about objects within the acquiredimage.

The acquired image information regarding the robust signature of theacquired image may be the robust signature of the acquired image.

The acquired image information regarding the robust signature of theacquired image may be a cortex representation of the signature.

The method may include acquiring first vehicle images by the firstvehicle; extracting, from the first vehicle acquired images, informationabout objects within the first vehicle acquired images; and preformingthe driving related operation of the first vehicle based on theinformation about objects within the acquired image and based on theinformation about objects within the first vehicle acquired images.

The image information may represent data regarding neurons of a neuralnetwork, of the second vehicle, that fired when the neural network wasfed with the acquired image.

The extracting, of the information about objects within the acquiredimage may include (a) comparing the signature of the acquired image toconcept signatures to provide comparison results; each concept signaturerepresents a type of objects; and (b) determining types of objects thatmay be included in the acquired image based on the comparison results.

FIG. 37 illustrates one vehicle that updates another vehicle—especiallywith objects that are not currently seen by the other vehicle—therebyallowing the other vehicle to calculate his driving path based (also) onthe object it currently does not see.

Concept Update

A concept structure includes multiple signatures that are related toeach other and metadata related to these signatures.

The concepts may be updated by removing signatures—and even parts of thesignatures that are more “costly” to keep than maintain.

The “cost” may represent one or more factors such as false detectionprobability, a robustness of the concept, an accuracy of the concept,and the like.

One or more vehicles may decide to update a concept—and then send theupdate (or an indication about the update) to other vehicles—thereforeimproving the detection process by using updated concepts.

FIG. 38 illustrates a method 2710.

Method 2710 may include steps 2712, 2714 and 2726.

Method 2710 is for concept update.

Method 2710 may include:

-   -   Step 2712 of detecting that a certain signature of an object        causes a false detection (see, for example FIGS. 39 and 40). The        certain signature belongs to a certain concept structure that        may include multiple signatures. The false detection may include        determining that the object may be represented by the certain        concept structure while the object may be of a certain type that        may be not related to the certain concept structure. For        example—a concept of a pedestrian may classify (by error) a mail        box as a pedestrian.    -   Step 2714 of searching for an error inducing part (see, for        example FIGS. 41-43) of the certain signature that induced the        false detection.    -   Step 2715 of determining whether to remove the error inducing        part of the certain signature. This may involve calculating a        cost related to a removing the error inducing part from the        concept structure—and removing the the error inducing part when        the cost may be within a predefined range.    -   Step 2726 of removing (see, for example FIG. 44) from the        concept structure the error inducing part to provide an updated        concept structure.

Each signature may represent a map of firing neurons of a neuralnetwork.

Step 2714 may include:

-   -   Generating or receiving a test concept structure that        includes (a) first signatures of images that may include one or        more objects of the certain type (that should not belong to the        concept), and (b) second signatures of second images that may        include one or more objects of a given type that may be properly        associated with the concept structure. The images of both types        may be selected in any manner—for example may be randomly        selected.    -   Comparing the certain signature to the test concept structure to        provide matching first signatures and matching second        signatures.    -   Comparing the matching first signatures and matching second        signatures to find parts that causes false errors and parts that        result in positive detection.    -   Defining the error inducing part of the certain signature based        on an overlap between the matching parts of the certain        signature.

The updated concept may be shared between vehicles.

Determination of Score Driver Behavior

Human drivers' behavior and driving skills are in many cases causeaccidents or lead to dangerous situations. For example, it is notuncommon for a driver not to stop at a stop sign, cross the crossing ina red light, cross a solid white line etc—situations that endanger thesurrounding road users. Unless a police officer is around, theseviolations go unnoticed

FIG. 45 shows illustration of an exemplary driver behavior scoringsystem 5000 (hereinafter referred to also as “system 5000”) constructedand operative in accordance with embodiments described herein. Asdescribed herein, system 5000 may be operative to process sensor andbehavior data to generate a score for the driver in order to warn otherdrivers if there is a dangerous driver around, automatically reportviolations to the police, inform insurance companies to allow reducedfees for good drivers, inform parents if a young driver performs anyviolations etc.

System 5000 comprises vehicle 2100, expected behaviors database 2103,drivers' score DB 2107, and driver behavior scoring server 2102 whichmay be configured to communicate with each other over a communicationsnetwork such as, for example, the Internet, It will be appreciated thatthe embodiments described herein may also support the configuration ofvehicle 2100 with an assisted (or “semi-autonomous”) driving systemwhere in at least some situations a human driver may take control ofvehicle 2100.

In accordance with the exemplary embodiment of FIG. 45, vehicle 2100 maybe configured with at least one sensor 2101 to provide information abouta current driving environment as vehicle 2100 proceeds along roadway2105. It will be appreciated that while sensor 2101 is depicted in FIG.45 as a single entity, in practice, as will be described hereinbelow,there may be multiple sensors 2101 arrayed on, or inside of, vehicle2100. In accordance with embodiments described herein, sensor(s) 2101may be implemented using a conventional camera operative to captureimages of roadway 2105 and objects in its immediate vicinity, eg a stopsign 2104A.

It will be appreciated that sensor 2101 may be implemented using anysuitable imaging technology instead of, or in addition to, aconventional camera. For example, sensor 2101 may also be operative touse infrared, radar imagery, ultrasound, electro-optics, radiography,LIDAR (light detection and ranging), etc. Furthermore, in accordancewith some embodiments, one or more sensors 2101 may also be installedindependently along roadway 2015, or inside other vehicles, 2106, whereinformation from such sensors 2101 may be provided to vehicle 2100and/or obstacle avoidance server 2102 as a service. The expectedbehaviors database 2103 holds the matrix of all possible traffic lawreference points 2104, and the expected driver actions respective ofeach reference points. For example, for a reference point of a “yield”sign, the expected behaviors database will include the “slowdown” and a“stop” actions.

In accordance with the exemplary embodiment of FIG. 45, traffic lawreference points 2104A and 2104B (collectively referred to hereinafteras reference points 2104) may be located along roadway 2105. Forexample, reference point 2104A is depicted as a stop sign, and referencepoint 2104B is depicted as a stop line marking on the road. Inoperation, sensor 2101 may capture images of reference points 2104. Theimages may then be processed by the driver scoring system 5000 invehicle 2100 to provide information about the quality of driving andtraffic violations.

FIG. 46 shows a flowchart of an exemplary driver behavior matchingprocess 6600 (hereinafter also referred to as process 6600) to beperformed by driver behavior scoring system during driving session forvehicle 2100.

In accordance with some embodiments described herein, system 5000 mayreceive (6601) a sensor data (e.g. image or video), use signalprocessing and recognition algorithms to analyze the sensor data (6602),and check if any of the reference points 2104 were detected by thealgorithm (6605).

Step 6603 includes receiving behavioral information of the vehicle whiledriving, such as location, speed, direction of propagation, damping ashock by a shock absorber, slowing the vehicle by the breaks, stoppingthe vehicle, turns of the steering wheel, and the like. This can beobtained via a Can BUS, telematic system and the sensors. If necessary,the behavior data is being analyzed by 2102 to translate the output of2102 to the action taken by the driver (the car slows down—the driverhit the brakes, if the vehicle changes direction—the driver turned thesteering wheel, if the car speeds up—the driver pushed the gas pedal,the accelerator, etc).

According to the detected reference point, the expected action by thedriver is loaded (6606) from expected behaviors database 2103. Then, theexpected action by the driver is matched to the actual action by thedriver (6607) and the results is recorded (6608) in the driver's scoreDB 2107. The driver action (both expected and actual) can be representedas an ID, and the matching 6607 is a simple ID matching, or it can alsohave properties, e.g. when slowing down, the property can be the rateand the amount of the slowdown, and in this case the match result 6607would be the distance between the expected and the actual actionproperties.

FIG. 47 shows a flowchart of an exemplary driver behavior grading/ratingprocess 6700 (hereinafter also referred to as process 6700). The matchbetween expected and the actual driver's action given an identifiedtraffic law reference point 2104 is recorded (6701) in the drivers'score DB 2107. Then the system checks if enough data exists in thesystem, for example if the user drove enough miles, if no, the systemkeeps collecting data and recording events without generating a scoreyet. Once enough data is collected the system 5000 triggers a gradingprocess 6700. This process can be triggered not only by a positivematching event of violation, but trigger itself every x number of miles,every x positive identifications of reference points etc.

Any reference in the specification to a method should be applied mutatismutandis to a system capable of executing the method and should beapplied mutatis mutandis to a non-transitory computer readable mediumthat stores instructions that once executed by a computer result in theexecution of the method.

Any reference in the specification to a system and any other componentshould be applied mutatis mutandis to a method that may be executed by asystem and should be applied mutatis mutandis to a non-transitorycomputer readable medium that stores instructions that may be executedby the system.

Any reference in the specification to a non-transitory computer readablemedium should be applied mutatis mutandis to a system capable ofexecuting the instructions stored in the non-transitory computerreadable medium and should be applied mutatis mutandis to method thatmay be executed by a computer that reads the instructions stored in thenon-transitory computer readable medium.

Any combination of any module or unit listed in any of the figures, anypart of the specification and/or any claims may be provided. Especiallyany combination of any claimed feature may be provided.

Any reference to the term “comprising” or “having” should be interpretedalso as referring to “consisting” of “essentially consisting of”. Forexample—a method that comprises certain steps can include additionalsteps, can be limited to the certain steps or may include additionalsteps that do not materially affect the basic and novel characteristicsof the method—respectively.

The invention may also be implemented in a computer program for runningon a computer system, at least including code portions for performingsteps of a method according to the invention when run on a programmableapparatus, such as a computer system or enabling a programmableapparatus to perform functions of a device or system according to theinvention. The computer program may cause the storage system to allocatedisk drives to disk drive groups.

A computer program is a list of instructions such as a particularapplication program and/or an operating system. The computer program mayfor instance include one or more of: a subroutine, a function, aprocedure, an object method, an object implementation, an executableapplication, an applet, a servlet, a source code, an object code, ashared library/dynamic load library and/or other sequence ofinstructions designed for execution on a computer system.

The computer program may be stored internally on a computer programproduct such as non-transitory non-transitory computer readable medium.All or some of the computer program may be provided on non-transitorycomputer readable media permanently, removably or remotely coupled to aninformation processing system. The non-transitory computer readablemedia may include, for example and without limitation, any number of thefollowing: magnetic storage media including disk and tape storage media;optical storage media such as compact disk media (e.g., CD-ROM, CD-R,etc.) and digital video disk storage media; nonvolatile memory storagemedia including semiconductor-based memory units such as FLASH memory,EEPROM, EPROM, ROM; ferromagnetic digital memories; MRAM; volatilestorage media including registers, buffers or caches, main memory, RAM,etc. A computer process typically includes an executing (running)program or portion of a program, current program values and stateinformation, and the resources used by the operating system to managethe execution of the process. An operating system (OS) is the softwarethat manages the sharing of the resources of a computer and providesprogrammers with an interface used to access those resources. Anoperating system processes system data and user input, and responds byallocating and managing tasks and internal system resources as a serviceto users and programs of the system. The computer system may forinstance include at least one processing unit, associated memory and anumber of input/output (I/O) devices. When executing the computerprogram, the computer system processes information according to thecomputer program and produces resultant output information via I/Odevices.

In the foregoing specification, the invention has been described withreference to specific examples of embodiments of the invention. It will,however, be evident that various modifications and changes may be madetherein without departing from the broader spirit and scope of theinvention as set forth in the appended claims.

Moreover, the terms “front,” “back,” “top,” “bottom,” “over,” “under”and the like in the description and in the claims, if any, are used fordescriptive purposes and not necessarily for describing permanentrelative positions. It is understood that the terms so used areinterchangeable under appropriate circumstances such that theembodiments of the invention described herein are, for example, capableof operation in other orientations than those illustrated or otherwisedescribed herein.

Those skilled in the art will recognize that the boundaries betweenlogic blocks are merely illustrative and that alternative embodimentsmay merge logic blocks or circuit elements or impose an alternatedecomposition of functionality upon various logic blocks or circuitelements. Thus, it is to be understood that the architectures depictedherein are merely exemplary, and that in fact many other architecturesmay be implemented which achieve the same functionality.

Any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality may be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected,” or“operably coupled,” to each other to achieve the desired functionality.

Furthermore, those skilled in the art will recognize that boundariesbetween the above described operations merely illustrative. The multipleoperations may be combined into a single operation, a single operationmay be distributed in additional operations and operations may beexecuted at least partially overlapping in time. Moreover, alternativeembodiments may include multiple instances of a particular operation,and the order of operations may be altered in various other embodiments.Also for example, in one embodiment, the illustrated examples may beimplemented as circuitry located on a single integrated circuit orwithin a same device. Alternatively, the examples may be implemented asany number of separate integrated circuits or separate devicesinterconnected with each other in a suitable manner.

Also for example, the examples, or portions thereof, may implemented assoft or code representations of physical circuitry or of logicalrepresentations convertible into physical circuitry, such as in ahardware description language of any appropriate type.

Also, the invention is not limited to physical devices or unitsimplemented in non-programmable hardware but can also be applied inprogrammable devices or units able to perform the desired devicefunctions by operating in accordance with suitable program code, such asmainframes, minicomputers, servers, workstations, personal computers,notepads, personal digital assistants, electronic games, automotive andother embedded systems, cell phones and various other wireless devices,commonly denoted in this application as ‘computer systems’.

However, other modifications, variations and alternatives are alsopossible. The specifications and drawings are, accordingly, to beregarded in an illustrative rather than in a restrictive sense.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim. The word ‘comprising’ does notexclude the presence of other elements or steps then those listed in aclaim. Furthermore, the terms “a” or “an,” as used herein, are definedas one or more than one. Also, the use of introductory phrases such as“at least one” and “one or more” in the claims should not be construedto imply that the introduction of another claim element by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim element to inventions containing only one suchelement, even when the same claim includes the introductory phrases “oneor more” or “at least one” and indefinite articles such as “a” or “an.”The same holds true for the use of definite articles. Unless statedotherwise, terms such as “first” and “second” are used to arbitrarilydistinguish between the elements such terms describe. Thus, these termsare not necessarily intended to indicate temporal or otherprioritization of such elements. The mere fact that certain measures arerecited in mutually different claims does not indicate that acombination of these measures cannot be used to advantage.

While certain features of the invention have been illustrated anddescribed herein, many modifications, substitutions, changes, andequivalents will now occur to those of ordinary skill in the art. It is,therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the invention.

We claim:
 1. A method for finding a trigger for human intervention in acontrol of a vehicle, the method comprises: receiving, from a pluralityof vehicles, and by an I/O module of a computerized system that islocated outside the plurality of vehicles, visual information acquiredduring situations that are suspected as situations that require humanintervention in the control of the vehicle; determining, based at leaston the visual information, what amounts to the trigger for humanintervention, wherein the human intervention comprises handing over acontrol of a vehicle from an autonomous driving manager of the vehicleto a human driver; and transmitting, by the computerized system, to thevehicle of the trigger visual information that represents the trigger;receiving by a processing circuitry of the vehicle the trigger visualinformation; acquiring, by a visual sensor of the vehicle, images of anenvironment of the vehicle; searching, by the processing circuit of thevehicle, images of the environment of the vehicle for a situation thatis identified by the trigger visual information; and when finding thesituation that is identified by the trigger visual information thentriggering the human intervention in the control of the vehicle.
 2. Themethod according to claim 1 wherein the determining of what amounts tothe at least one trigger for human intervention is based on an outcomeof an unsupervised learning process.
 3. The method according to claim 1wherein the determining of what amounts to the at least one trigger forhuman intervention is responsive to at least one human interventionpolicy of the at least one vehicle.
 4. The method according to claim 1wherein the determining of what amounts to the at least one trigger forhuman intervention comprises determining a complexity of the situation.5. The method according to claim 1 wherein the determining of whatamounts to the at least one trigger for human intervention comprisesdetermining a danger level associated with the situation.
 6. The methodaccording to claim 1 wherein the determining what amounts to isresponsive to statistics of maneuvers executed by different vehiclesduring a same situation that is suspected as a situation that requireshuman intervention in the control of at least one of the plurality ofvehicles.
 7. The method according to claim 1 comprising generating orreceiving movement information of entities included in the visualinformation; and wherein the determining of what amounts to the at leastone trigger is also responsive to the movement information.
 8. Themethod according to claim 7 wherein the movement information representsentity movement functions of the entities, wherein the method comprisesestimating, based on the entity movement functions, a future movement ofthe entities.
 9. The method according to claim 1 wherein the determiningis also based on verification information indicative of whether amaneuver related to a situation that is suspected as requiring humanintervention in the control of a vehicle is actually related to asituation that actually requires human intervention in the control ofthe vehicle.
 10. A non-transitory computer readable medium that storesinstructions for: receiving, from a plurality of vehicles, and by an I/Omodule of a computerized system that is located outside the plurality ofvehicles, visual information acquired during situations that aresuspected as situations that require human intervention in a control ofthe vehicle; determining, based at least on the visual information, ofwhat amounts to the at least one trigger for human intervention, whereinthe human intervention comprises handing over a control of a vehiclefrom an autonomous driving manager of the vehicle to a human driver; andtransmitting, by the computerized system, to the vehicle of the, triggervisual information that represents the trigger; receiving by aprocessing circuitry of the vehicle the trigger visual information;acquiring, by a visual sensor of the vehicle, images of an environmentof the vehicle; searching, by the processing circuit of the vehicle,images of the environment of the vehicle for a situation that isidentified by the trigger visual information; and when finding thesituation that is identified by the trigger visual information thentriggering the human intervention in the control of the vehicle.
 11. Thenon-transitory computer readable medium according to claim 10 whereinthe determining of what amounts to the at least one trigger for humanintervention is executed in an unsupervised manner.
 12. Thenon-transitory computer readable medium according to claim 10 whereinthe determining of what amounts to the at least one trigger for humanintervention is responsive to at least one human intervention policy ofthe at least one vehicle.
 13. The non-transitory computer readablemedium according to claim 10 wherein the determining of what amounts tothe at least one trigger for human intervention comprises determining acomplexity of the situation.
 14. The non-transitory computer readablemedium according to claim 10 wherein the determining of what amounts tothe at least one trigger for human intervention comprises determining adanger level associated with the situation.
 15. The non-transitorycomputer readable medium according to claim 10 wherein the determiningof what amounts to the at least one trigger for human intervention isresponsive to statistics of maneuvers executed by different vehiclesduring a same situation that is suspected as a situation that requireshuman intervention in the control of at least one of the plurality ofvehicles.
 16. The non-transitory computer readable medium according toclaim 10 that stores instructions for generating or receiving movementinformation of entities included in the visual information; and whereinthe determining of what amounts to the at least one trigger is alsoresponsive to the movement information.
 17. The non-transitory computerreadable medium according to claim 16 wherein the movement informationrepresents entity movement functions of the entities; wherein thenon-transitory computer readable medium stores instructions forestimating, based on the entity movement functions, a future movement ofthe entities.
 18. The non-transitory computer readable medium accordingto claim 17 that stores instructions for wherein the determining is alsobased on verification information indicative of whether a maneuverrelated to a situation that is suspected as requiring human interventionin the control of a vehicle is actually related to a situation thatactually requires human intervention in the control of the vehicle. 19.The method according to claim 1 wherein the trigger visual informationis generated based on visual information acquired by at least one visualsensor during an execution of at least vehicle maneuver that issuspected as being an obstacle avoidance maneuver.